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Abstract 



Spiked Models in Wishart Ensemble 

A dissertation presented to the Faculty of 
the Graduate School of Arts and Sciences of 
Brandeis University, Waltham, Massachusetts 

by Dong Wang 

The spiked model is an important special case of the Wishart ensemble, and a natu- 
ral generalization of the white Wishart ensemble. Mathematically, it can be defined 
on three kinds of variables: the real, the complex and the quaternion. For practi- 
cal application, we are interested in the limiting distribution of the largest sample 
eigenvalue. 

We first give a new proof of the result of Baik, Ben Arous and Peche for the 
complex spiked model, based on the method of multiple orthogonal polynomials by 
Bleher and Kuijlaars. Then in the same spirit we present a new result of the rank 1 
quaternionic spiked model, proven by combinatorial identities involving quaternionic 
Zonal polynomials {a — 1/2 Jack polynomials) and skew orthogonal polynomials. 

We find a phase transition phenomenon for the limiting distribution in the rank 
1 quaternionic spiked model as the spiked population eigenvalue increases, and rec- 
ognize the seemingly new limiting distribution on the critical point as the limiting 
distribution of the largest sample eigenvalue in the real white Wishart ensemble. 

Finally we give conjectures for higher rank quaternionic spiked model and the real 
spiked model. 



V 



Contents 



Abstract |v] 

1 Introduction [1] 

1.1 Wishart distribution [1] 

1.2 Spiked models of Wishart ensemble [7] 

1.3 (Generalized) Zonal polynomials [TT] 

1.4 Statement of Results [TT] 

1.5 Structure of the thesis [23] 

2 Complex spiked model [25] 

2.1 Determinantal joint p. d.f. formula [25] 

2.2 Multiple orthogonal Laguerre polynomials [32] 

2.3 Proof of theorem O M 



3 Rank 1 quaternionic spiked model [53] 

3.1 The joint distribution function [22] 

3.2 The determinantal formula [59] 

3.3 S4^{x,y) in terms of Laguerre polynomials [63] 

3.4 Proof of theorem [L2] UQ\ 

4 Phase transition phenomenon 1821 

4.1 Rank 1 complex spiked model [82] 

4.2 Rank 1 quaternionic spiked model [13 

4.3 Conjectures of phase transition in quaternionic and real spiked models [HI] 

5 Asymptotic analysis [§3] 

5.1 Asymptotics of ^/^(p + g^), i'r'ip + ipip + qr]) axid ipriij) + qri) when 

a,/ < 7-1 [91 

5.2 Asymptotics of ipr'ip + qi) and ipr'ip + qv) when a^/ = 7"^ II 101 

5.3 Asymptotics of + g^), '^/'r' (P + '?0 5 i^ip + qv) s.nd ipr'{p + qv) when 

agi < a 11151 

5.4 Asymptotics of Laguerre polynomials and related functions 11281 



vi 



Chapter 1 
Introduction 



1.1 Wishart distribution 

The Wishart distribution is a multivariate generahzation of the distribution. The 
distribution is defined by the normal distribution. The normal distribution is 
well known as a distribution of one real variable. However, in this thesis we study 
statistics of three kinds of variables: real, complex and quaternion. Thus we begin 
the thesis with a review of the normal distribution of all the three kinds of variables. 
All results for real variables are standard, see e.g. [22]; for complex variables, see [T3j : 
for quaternion variables, see 

1.1.1 Normal distribution 

The normal distribution of a real variable with mean /i and variance cr^ is defined by 
the probability density function (p.d.f.) 




(1.1) 



1 
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When we study the distribution of a complex variable z, we can view it as two 
possibly dependent real variables x and y, which are its real and imaginary parts: 
z = X + iy. The definitions of mean and variance is similar to those of real variables: 

E{z)=E{x) + E{y), (1.2) 
var(^) =E((z - E(z))(z - E(z))) = (Eix^) - E^ix)) + (Eiy') - E^iy)) 

= var(x) + var(?/). (1.3) 

The normal distribution of a complex variable z with mean /i and variance is 
defined by the p.d.f. 

Piz) = —e ^ = — e-^, 1.4 

SO that X and y are independent real variables in normal distribution with means 
5R(/i) and Q'(/i) respectively, and identical variance cr^/2. 

A quaternion variable u has 4 real parts: u = x + iy+jz + kw. The definitions 
of mean and variance is similar to (|L2[ and [] 



E(u) = E{x) + iE{y) + j E{z) + kE{w), (1.5) 



var(M) = E{{u — E{u)){u — E{u))) = var(x) + var(?/) + Yai{z) + var(ty). (1.6) 

The normal distribution of a quaternionic variable m with mean /i and variance is 
defined by the p.d.f. 

^H = ^^^"^' (1-7) 

7r"^(T*/4 

so that the 4 parts, x, y, z and w are independent real variables in normal distribution 
with means the corresponding parts of fi and identical variance cr^/4. 



^In this thesis we use the same notations of conjugation and norm for both complex and quater- 
nion variables. For quaternions, x + iy + jz + kw = x — iy — jz — kw and \x + iy + jz + kw\ = 
y^x"^ + y"^ + z^ + w^. 
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1.1.2 distribution 

The distribution, like the normal distribution, can be defined for all the three 
kinds of variables. Let x be a random variable in normal distribution with mean 
and variance a^, and we take k independent measurements of x, with results xi, . . . , 
Xk, which are accordingly random variables with identical independent (i.i.d.) (0, cr^) 
normal distribution. If we define the random variable 



then s/(7^ is in the xl distribution, i.e., chi-square distribution with parameter k. 

No matter for what kind of variables xl distribution is defined, it is the distribution 
of a real random variable with support [0, cxo). However, the p.d.f.s of Xk ^lot 
identical for the three kinds of variables: 

• The p.d.f. of Xk ^r a real variable: 



k 




(1.8) 



P{x) 



(1/2)^/2 , 
— — - — a;2 

r(fc/2) 



(1.9) 



• The p.d.f. of Xk ^ complex variable: 



P{x) 



1 



(1.10) 



m 



X' 



• The p.d.f. of Xk ^r a quaternion variable: 



P{x) 



r{2k) 



X' 



(1.11) 
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1.1.3 Multivariate normal distribution 
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The normal distribution has multivariate versions for all the three kinds of variables. 
The n-variate random variable is represented by an n-dimensional column vector 
) , where possibly correlated random variables. The variable 

in the multivariate normal distribution has a mean /i, which is an ra-dimensional 
column vector, and a covariance matrix S, which is a positive defined symmetric, 
Hermitian or quaternionic Hermitian matrix depending on which kind of variables 
we consider. They are given by 

/^i=E(x,), (1.12) 
=cov(xi,Xj), (1.13) 



where for any kind of variables, 



cov(a:, y) = E((x - E(x))(y - E(2/))), (1.14) 

with the overline meaning conjugation for complex and quaternion variables and the 
identity for real variables. 

Conversely, yU and S determines the p.d.f. of X, by slightly different formulas for 
the three kinds of variables: 

• The p.d.f. of an n-variate real normal variable: 

(27r)"/2det(E)V2^ • ^^'^^^ 

^The definition of a quaternionic Hermitian matrix {aij)i<ij<N is similar to that of a Hermitian 
matrix: (1) Diagonal real numbers. (2) Strictly upper-triangular entries a^j's with i < j 

are arbitrary quaternions. (3) = ITJi for strictly lower triangular entries aij with i > j. 
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• The p.d.f. of an n-variate complex normal variable: 



^ ^ 7r"det(E) ^ ^ 



• The p.d.f. of an n-variate quaternion normal variable: 



(7r/2)2-det(S)2'' ■ ^"--^'^ 



We need to take notice that the determinant is not well defined for quaternion 
matrices due to the noncommutativity of quaternions. Here since E is a quaternionic 
Hermitian matrix with positive real eigenvalues iTi, . . . , (T„, we define 

n 

det(S) = J]a,. (1.18) 
1.1.4 Wishart distribution 

Now we can define the Wishart distribution, which is similar to the distribution. 
Let X be an A^-variatc random variable in the normal distribution with mean and 
covariance matrix E, and we take M independent measurements of X with result Xi, 
. . . , Xm, which are accordingly random variables with i.i.d. (0, E) normal distribution. 
If we let the N x M matrix X be the juxtaposition of Xj^s: X = [Xi :,...,: Xm), 
then we say that the A^^-variate random variable 

S = ^XX^ (1.19) 



has the Wishart distribution Wf^{M^ E) with M degrees of freedom and covariance 
matrix E. 

Given M and E, we have exphcit formulas of p.d.f.s of S for all the three kinds of 
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variables. Since the spaces oi NxN real symmetric, Hermitian and quaternionic Her- 
mitian matrices are Euclidean spaces with dimensions N{N — N'^ and N(2N — 1) 
respectively, we take the usual definition of the measure: For S = {spq) real symmetric, 
dS = Ili<q<p<N dspq- For S = (spq) Hermitian, dS = Hii dsrr Ili<q<p<N d^Spqd^Spq. 
For S — (spq) quaternionic Hermitian and Spg — Xpg+iypq+jZpq+kWpq for off-diagonal 
entries, dS = Y[r=i ^rr ni<g<p<Ar dxpqdypqdzpqdwpq. To make the support of S be the 
full positive definite cone of the space of real symmetric matrices, Hermitian matrices 
or quaternionic Hermitian matrices, we require that M > N. 

• The p.d.f. of the real Wishart distribution Wn{M, E): 



P{S)dS 



1 



2MJV/27,iv(iv-i)/4(det E)A^/2 n;=i r((M - j + 1) /2) 

g-lTV(E-5)(^g^^)(M-iV-l)/2^^_ (1.20) 



• The p.d.f. of the complex Wishart distribution Wn{M, E): 



P{S)dS 



1 



e-^(^"^^)(det5)^-^d5. 



7riv(iv-i)/2(det E)^ nj=i r(M - j + 1) 



(1.21) 



• The p.d.f. of the quaternion Wishart distribution Wn{M, E): 



P{S)dS 



7r^(^-i)(det E)2A^ 11^^^ r(2(M -j + 1)) 



g-2SRlV(E-i5)(^g^^)2(M-iV)+l^^_ (1.22) 



Here we should take note that although Tr(E~^5') is automatically real for E and S 
to be real symmetric or complex Hermitian matrices, we need to take the real part 
explicitly in the quaternionic case due to the noncommutativity. 
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1.2 Spiked models of Wishart ensemble 

In statistics, the eigenvalues ai, . . . , aN of the covariance matrix S in the multivariate 
normal distribution are called population eigenvalues. They are of importance in 
principal component analysis. 

Let X be a centralized A^-variate random variable, which means its mean is 0. 
Under the assumption that X has the (multivariate) normal distribution, how can we 
determine its population eigenvalues by results of measurements? 

In the = 1 case, it is equivalent to find the variance a^. If we make k indepen- 
dent measurements and get results xi, . . . , Xk, we have the random variable s defined 



in (1.8), and it is easy to find that s/k approaches o"^ almost surely as k ^ oo. 



For general N, if we make M measurements and get results Xi, . . . , X^r, we have 



the N X N random matrix S defined in (1.19), which is called the sample covariance 



matrix in statistics. The multivariate counterparts of s are the eigenvalues Ai, . . . , 
Xn of S, which are called sample eigenvalues in statistics. A celebrated result of 
Anderson [3] states that if M X, the sample eigenvalues are good approximations 
of the population eigenvalues. 

However, if M is not much greater than N, say, both N and M are large, and 
M/N = 7^ > 1, then the sample eigenvalues fail to approximate the population 
eigenvalues. For example, if a/s are identically 1, Marcenko and Pastur proved [20] 



Proposition 1.1 (Marcenko-Pastur law). When = I, as M,N — > oo such that 
M/N — 7^ > 1, the limiting density of the sample eigenvalues Xi in the complex 
Wishart ensemble is given by 

^M>^,\X,<x}^H{x), (1.23) 
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where (bi = (1 — 7 and 62 = (1 + 7 ^Y) 







X < b 



Hi^)={Z£iVit-bi)ib2-t) b,<x<b,. 



(1.24) 




v 



Actually the proof of Marcenko and Pastur is only for real and complex vari- 
ables. However, their proof can be transplanted to the quaternion case without much 
difficulty. 

In this thesis, if the covariance matrix S is given, we call the distribution of the 
sample eigenvalues Ai, the Wishart ensemble. It is easy to see that the 

Wishart ensemble is completely determined by the population eigenvalues cxi, . . . , (Tat 
and the number of measurements M. The Marcenko-Pastur law gives the density of 
the sample eigenvalues in the S = / Wishart ensemble, which is commonly called the 
white Wishart ensemble 

The general problem of getting information about a^'s from properties of the 
Wishart ensemble is far too difficult. We begin with the hypothesis testing problem: 
For E and S' with different population eigenvalues, can we tell the difference between 
the corresponding Wishart ensembles? In the simplest case, the eigenvalues of S are 
identically 1, and we get the white Wishart ensemble; most eigenvalues of S' are 1 
while the other r sample eigenvalues are 1 + ai, . . . , 1 + a,.; with a^-'s real numbers 
greater than —1, and we call the corresponding Wishart ensemble the spiked model 
of rank r. Now the question is: Can we tell the spiked model from the white Wishart 
ensemble? 

The density of sample eigenvalues fails to detect the difference, since the proof of 

■^In the random matrix theory Hterature, the real white Wishart ensemble is called the Laguerre 
orthogonal ensemble (LOE), and the complex and quaternionic white Wishart ensembles are called 
the Laguerre unitary ensemble (LUE) and the Laguerre symplectic ensemble (LSE). 
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Marcenko and Pastur implies the stronger result [20] : 



Proposition 1.2. Let r be a fixed positive integer and ai, . . . , Or fixed real numbers 
greater than —1. When M,N —>■ oo such that M/N — * 7^ > 1, and the population 
eigenvalues are 1 + ai, . . . , 1 + and all others identically 1, for all the three kinds 
of variables, the limiting density of the sample eigenvalues Xj 's in the rank k spiked 
model is given by 

N' 



<x} ^ H{x), (1.25) 



where H{x) is defined in (1.24), the same as that in the white Wishart ensemble. 



However, if some population eigenvalues are large, the limiting distribution of 
the largest sample eigenvalue may change. First, we have a complete result for the 
limiting distribution of the largest eigenvalue in the white Wishart ensemble. Unlike 
the limiting density, these limiting distributions for the three kinds of variables are 
different [9], p], [IS]. 

Proposition 1.3 (GOE, GUE and GSE Tracy-Widom distributions). When S = /, 
as M,N — > oo such that M/N — 7^ > 1, the largest sample eigenvalue max(A) 
approaches (1 + 7~^)^ almost surely for all the three kinds of variables. However, the 
limiting distributions for these three kinds of variables are different. 

• For real variables, the limiting distribution o/max(A) is the GOE Tracy-Widom 
distribution, after proper rescaling: |^ 

Jim P ((max(A) - (1 + 7^^)^) " j^f^, < t) = Fcoe(T). (1.26) 



• For complex variables, the limiting distribution o/ max(A) is the GUE Tracy- 

^By AI — > oo, we mean "M, N oo and M/N ~ 7^. 
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Widom distribution, after proper resettling: 



Jirn^P (^(max(A) - (1 + 7-^)') ■ ^^T^ < ^ ' = ^GUE(r). (1.27) 



For quttternion vttrittbles, the limiting distribution o/max(A) is the GSE Trttcy- 
Widom distribution, ttfter proper resettling: 

lim P f (max(A) - (1 + 7^')') ■ < t] = Fgse(T). (1.28) 



We have explicit formulas for these probability functions [28], [29] : 



Fgoe(0 =e-5/r(^-«)'?'(^)'^^e~2/r9W^^, (1.29) 

Fgue(0 =e-r (-)'^-, (1.30) 

where g(a;) is a solution to the Painleve equation 

q{x) = xq{x) + 2q^{x), (1.32) 

with 

q{x) ~ Ai(x) as x ^ +00. (1.33) 



Here Ai(x) is the Airy function, whose definition will be given in (1.62). 

It is worth noticing that although all previous statements for the three kinds of 
variables are parallel, the three-fold symmetry breaks down and our statements for 
the three kinds of variables are going to bifurcate. Despite their similarity in analytic 
form, -Fgue is most naturally defined by a Fredholm determinant while -Fgoe and -Fgse 
are most naturally defined by Fredholm Pfaffians, and the derivations of -Fgoe and 
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Fgse are somehow similar to each other, and requires more work than tlie derivation 

of fgue m- 

Now the goal is to get the limiting distribution of the largest eigenvalue in spiked 
models of the three kinds of variables. For the complex variables, Baik, Ben Arous 
and Peche got the complete result for any finite rank /c [6]. In this thesis we derive 
their result by a different approach and are going to get that limiting distribution for 
the rank 1 quaternionic spiked model. It is desirable to find the counterpart limiting 
distribution for the rank 1 real spiked model. However, it seems that the symmetry 
between real and quaternion variables breaks again, and such a result is unattainable 
by the method in this thesis. 



1.3 (Generalized) Zonal polynomials 



From the p.d.f.s of the Wishart distributions ( 1.20 )-( 1.22) for the three kinds of 
variables, we can get the p.d.f.s of sample eigenvalues A = (Ai, . . . , Aat) by the Weyl 
integration formula, or more directly, by calculating Jacobians. Since we do not need 
explicit formulas of the normalization constants, we simply write "C" from now on. 
The derivation for real variables is in [22] 

• The p.d.f. of the sample eigenvalues in the real Wishart ensemble: 

N 

P(A) = i|v(A)| rr Af / e-f T^(^-^^^«-^)rfO. (1.34) 

^ Jo(N) 



The p.d.f. of the sample eigenvalues in the complex Wishart ensemble: 



1 ^ r 
P(A) = ±-V{Xf n Af / e-^^T'^^^-'^^^-'^rft/. (1.35) 

C fJl JuiN) 
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• The p.d.f. of the sample eigenvalues in the quaternion Wishart ensemble: 

P(A) = 1f(A)4 fr xf'-""^^' [ e-^^'^^^^^-'^'^^-'^dQ. (1.36) 

C fJl Jsp(N) 



Here V{X) = ni<i<j<Ar('^«~'^j) is the Vandermonde, and the integrals in (1.34)-(1.36) 
are over orthogonal, unitary and compact symplectic groups with Haar measures 
respectively. To go further in our analysis, we need to evaluate these integrals. 



It is well know among statisticians that we can expand the integral in (1.34) 
by Zonal polynomials [22]. To define Zonal polynomials, and their counterparts for 
complex and quaternion variables, we need some preliminary definitions. 

Definition 1.1. [19j A partition k of A; is a sequence k = {ki, K2, . . . , ki) where kj > 
are weakly decreasing and Yl]=i ~ ^- denote this hj k\- k. 

For example, k = (2, 2, 1) is a partition of 5. The number of nonzero parts of k is 
called the length of k, denoted as If we drop the weakly decreasing condition, 
we call the sequence a general partition. 

If K and K,' are two general partitions of k, we say k < k' ii for some index j, 
Ki = k'j for i < j and Kj < k'j. For example, 

(2,1,1,1) < (2,2,1) < (3,2). (1.37) 
If K h /c is a general partition with Z(/t) = /, we define the monomial of degree k 



x'' = x^'x';\..Xi' (1.38) 



and say x'^' is of higher weight than x'^ if k' > k. 

We need another definition of Laplacians Aj^, and [25j: 
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Definition 1.2. For N variables x = {xi, . . . , xn), we define 

^ ^2 ^ x^- d 
^X]^i^2:^ 7~3T7^7T' (1.39) 

j=l J i=l ■' ■' 

Now we can give a definition of Zonal polynomials and their counterparts, complex 
Zonal polynomials and quaternionic Zonal polynomials |18j . 

Definition 1.3. For variables x = {xi, . . . ,xn), a nonnegative integer k and a 
partition k \- k, we have the unique Zonal polynomial Zf^{x), complex Zonal poly- 
nomial Ck(x) and quaternionic Zonal polynomial Qk.{x), which are all symmetric, 
homogeneous polynomials of degree k in Xj's such that 

• The highest weight term in Zf.{x) (Ck(x), Qk{x)) is x'^. 

• Zk(x) (Ck(x), Qnix)) is an eigenfunction of the Laplacian (A^, Aj^). 

5^ Z,{x) = J2 C.{x) = J2 = (xi + ■ ■ ■ + xn)". (1.42) 

fchfc Khk K\-k 

Form the highest weight property of Zi^{x) (Ck(x), Q^{x)), we have 
Fact 1.1. For any and k\- k, 

Z^{x) = C,ix) = Q,ix) = if 1{k)>N. (1.43) 

Latter we apply the notation Zi^{X) to mean Zi^{xi^ . . . ,xn) if X is an x 
matrix with eigenvalues xi, . . . ,xn, and similarly for Ck(X) and Qk{X). 
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It can be derived from their eigenfunction property that [22], [IH], [19], [H] 

Proposition 1.4. Given X and Y to be N x N symmetric matrices, Hermitian 
matrices or quaternionic Hermitian matrices, and /at to he the N xN identity matrix, 
we have 



Z,{X)Z,{Y) 



I Z,{XOYO-')dO 

JO{N) ^k\J-N) 

[ C,{XUYU-')dU _^-(^)^'^(^) 

Ju(N) 



'UiN) 

^Q^{XQYQ-^)dQ 



Q,{X)Q^{Y) 



Sp{N) 



N) 



Then by (1.42) we have 



^pTriXOYO- 



'dO 



0{N) 



')dO=Y^ / Tt(XOYO'^] 

CO p 

Ef[ E / Z.iXOYO-')dO 
TT^ i^- Jo{N) 

Z,{X)Z,{Y) 



fc=o - .hfc -Jow 

1(k)<N 



A;=0 Khfc 
1{k)<N 



N 



and similarly 



/ 

JU(N) 



oo ^ 

fc=0 rehfc 
«(K)<Ar 



Sp(7V) 



A:=0 Khfc 
i(K)<Ar 



a(/7v) ' 

Q^{X)Q^{Y) 

QMn) 



1.44) 
1.45) 
1.46) 



(1.47) 



1.481 



1.49) 



Now we can state the series formulas for joint p.d.f.s for the three kinds of variables 
of the Wishart ensemble: 



Proposition 1.5. Let the centralized N-variate normal random variable have the 
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covariance matrix S with population eigenvalues ai, . . . , a^, and the number of 
measurements be M > N . The p.d.f. of the sample eigenvalues A = (Ai, . . . , Atv) in 
the Wishart ensemble is 

• For the real variable case 

P(A) = ^inA)inAf— 

1{k)<N 



For the complex variable case 



N 



1(k)<N 



For the quaternion variable case 



N 



^{-2Mf_ ^ g.K-\...,cr-^)g.(Ai,...,A;v) 

^ k\ 2^ QJIm) ■ ^ ■ 

fc=0 KhA: ^'"^ 

1{k)<N 



52) 



Remark 1.1. The Zonal, complex Zonal and quaternionic Zonal polynomials are Jack 
polynomials with the parameter a = 2, 1 and 1/2 [19j. In particular, complex Zonal 
polynomials (a = 1 Jack polynomials) are essentially Schur polynomials, see (2.1). 
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Although in formulas ( 1.50 )-( 1.52) we get rid of integrals over Lie groups, the 
number of degree k terms grows very fast as k increases, since the only restriction 
l{k) < N is rather weak when we consider the large A^. For the general Wishart 
ensemble these formulas are still impractical. However, they can be much more 
powerful in spiked models. 

Since in the spiked model, lots of cr^'s are identically 1, we can shift coordinates 



to make them identically 0, and then by fact 1.1, for rank r spiked model we need 
only consider (complex, quaternionic) Zonal polynomials with index /(/t) < r, since 
after the coordinate shift, N — r variables are and the A^- variable polynomials is 
equivalent to an r-variable one. For real variables the procedure is (eigenvalues of S 
are Ai, . . . , Xn, and eigenvalues of S are ai, . . . , ar, 1, . . . , 1) 



0{N) 

g- f Tr(/050-i)gf Tr({/-S-i)OSO-i)^(-, 
0{N) 

(1.53) 



i=l ■^O(N) 



^ ■• - {M/2f ^ Z.(^,...,^)Z.(Ai,...,A, 



3 = 1 k=0 fthfc ^ ' 

Z(K)<r 

since r eigenvalues of / — are tt^, • • • , tt^ and the other N — r eigenvalues are 
0. In this way we can simplify formulas for joint p.d.f.s of sample eigenvalues for the 
three kinds of variables in the spiked model: 

Proposition 1.6. Let the centralized N-variate normal random variable have the 
covariance matrix S with r population eigenvalues 1 + ai, . . . , 1 + and the other 
N — r population eigenvalues identically 1, and the number of measurements be M > 
N. The p.d.f. of the sample eigenvalues A = (Ai, . . . , Atv) in the rank r spiked model 
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is 



• For the real variable case 



m = ^\v{x)\fix. 



{M-N-l)/2 M^. 



OO 



l+ai ' 



. .., j^)Z«(Ai,...,A 



k=0 



fthfe 
l(K)<r 



;i.54) 



• For the complex variable case 



OO 



1^ u 2-^ 



f^)C'«(Ai,...,A7v) 



k=Q 



K\-k 
1(k)<N 



CM 



;i.55) 



• For the quaternion variable case 



1 ^ 
P(A) = ^V{Xyllxf''-''^^'e-'^^^ 



C 



f.(-2Mf ^ Q^(^^,...,,^^)Q^(X„...,X^) 

1(k)<N 



1.4 Statement of Results 



1.4.1 Complex spiked model 

To demonstrate the result, we need the language of Predholm determinant. If K{x, y) 
is the kernel of an integral operator from L^(M) to L^(M), then the Fredholm determi- 
nant of the integral operator, which we represent by the same notation as its kernel. 
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is [12] 



det {I -K{x,y)) 
1 



1 - 



1! 



K{xi, Xi)dxi + 



oo roo 



oo J — oo 



W. 



2! 

/r(a;i,xi) 

K(^Xn, Xi J 



K{xi,xi) K{xi,X2) 

K{x2,Xi) K{X2,X2) 

.. K{xi 



dxidx2 — . . 



K{Xfi^ Xf 



dxi . . . dxn + 



;i.57) 



Then we can state the theorem for the complex spiked model [6] 

Theorem 1.1. In the rank r complex spiked model, let non-trivial population eigen- 
values he 1 + ai < ■ ■ ■ < 1 + tts with multiplicities respectively ri, . . . , Ts, so that 

1. If —1 < as < 7~"^, then the distribution of the largest sample eigenvalue is the 



same as that of the complex white Wishart ensemble in proposition 1.3 



2. If as = '-f ^, then the limit and the fluctuation scale are the same as those of 
the complex white Wishart ensemble, but the distribution function is 



lim P max(A) — 



7 + 1 

7 



(7 + 1)^/3 



<T] =FGUEr-.(T). (1.58) 



3. If as = a > '-f ^, then the limit and the fluctuation scale are changed as well as 
the distribution function, which is a finite GUE distribution 



to^pf(ma.x(A)-(l + a) (l + +)) 



M 



'l + a)Jl 



< T 



GrXT). (1.59) 
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We need to explain the distribution function -FcuErs and Gr^. First, let us revisit 
the distribution -Fque, which can be defined by (We abbreviate the indicator function 
X[T,oo)(a;) as xix).) [28] 

Fgue(T) = det (1 - x(O^Airy(e, r])x{r])) , (1.60) 
where the kernel -f^Airy is given by the Airy function 

POO 

^Airy(e,^)= / Mix + t) Aiiy + t)dt, (1.61) 
Jo 



with fl] 

27V 



1 pooe''^/^ 3 

Ai(a;) = — / e'^^+'^sdz. (1.62) 

OOg57ri/6 



The integral sign J^^s^i/e nieans the integral is along an infinite arc from the direction 
^g57ri/6 direction ooe'^^^^. (We borrow the notation from the real projective 

geometry on MP^.) It is easy to see that as x — >■ +oo, |Ai(x)| faster than any 
exponential decay. The equivalency of the Fredholm determinant representation and 



the formula (1.30) is established by Tracy and Widom. 



FcvEt with t = 1, 2, . . . are variations of -Fgue, and are defined as [6] 



FcuEtiT) = det I 1 - xiO I ^Airy(e,r7) + J2^^'\0s^'\v) ] Xiv) ] , (1-63) 



where s*--^-* and t^^^ are variations of the Airy function, with 



1 r^^'^" ......3 1 



sO)(x)=— e''''+''^-—dz, (1.64) 

t(i)(x)=— e'^^'+'^^i-tzy-^dz, (1.65) 

2tt 



ooe 



57ri/6 
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and we require that the infinite arc for the integral of s^-^-* is below the pole z = 0. 
We can prove that as x ^ +oo, |t*--'''(x)| faster than any exponential decay, and 
gU) grows slower than any exponential growth. Especially, 



s(^\x) =1 - / Ai{t)dt, (1.66) 
tW(x) =Ai(x). (1.67) 



Gt with t = 1, 2, . . . are defined as 



Gt{T) = det 1^1 - x(e) (^g -^H,{i)e-'iH,{r,)e-''^'^ xiv) 



:i.68) 



where -ff/s are Hermite polynomials [27], with deg(ifj) = j and 

Hi{x)Hj{x)e-^ dx = j\V2i5ij. (1.69) 



Remark 1.2. Since Hq{x) = 1 is a constant, we can easily see that Gi is the Gaussian 

distribution ^ 

Gi(T) = [ -^e-'^dt. (1.70) 

Remark 1.3. We call -Fque, -^gue* and Gt distribution functions, because they are 
monotone increasing, as T — > +oo, the values of these functions approach 1, and as 
T ^ — oo, the values approach 0. Their monotonicity is ensured by their definitions. 
In the T —>■ +oo direction, we can easily verify the limit property from the deter- 
minantal representation. However, in the T —>■ —(yo direction, it is a more serious 
problem. 

Since Gt is defined by a finite rank kernel with relatively simple functions, we can 



verify limr^_oo = by direct calculation. For -Fque, the analytic formula (1.30) 



whose derivation from the determinantal formula (1.60) is highly non-trivial, yields 
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the limit result as T — > — oo. For -Fgue*, we refer to Baik's result [5]. 
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1.4.2 Rank 1 quaternionic spiked model 

To state the result for the rank 1 quaternionic spiked model, we need the definition 
of Fredholm determinant for a matrix integral operator. If K is an matrix integral 
operator from L^(]R)" to L^(]R)"', i.e., K = {Kij{x, y))i<i,j<n, with Kij{x, y) an integral 
operator from L^(]R)" to L^(]R)", then 



det(/-ir) = i + 5^(-ir E 

0<rj<m 



ri! . . . r„! 

m=l 0<r,<m 



/oo /-oo ' 1 ' n / \ 

... n ^^'^ ■ ■ ■ n ^4"^ 4'^)i<^<^. • (1-71) 
■oo -^-ooj^i \ ^<l<n / \<k,l<m 

Then we can state the theorem for the rank 1 quaternionic spiked model 
Theorem 1.2. In the rank 1 quaternionic spiked model, 

1. If —1 < a < 7~^, then the distribution of the largest sample eigenvalue is the 



same as that of the quaternionic white Wishart ensemble in proposition 1.3 



2. If a = ^ , then the limit and the fluctuation scale are the same as those of the 
quaternionic white Wishart ensemble, but the distribution function is 

7 + lV\ 7(2M)2/3 



Jhn^P max(A) -{—))■ < ^ 1 = ^--(^)- ^^''^^ 



3. If a > '-f ^, then the limit and the fluctuation scale are changed as well as the 
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distribution function, which is a Gaussian: 



Jto pf(max(A)- (a +l)(l + -!-)) 



V2M 



(a + l)Jl 



< T 



1 _^i , 
e 2 dt. 



;i.73) 



Here -Fqsei is a variation of -Fgse, and we first give a definition of -Fgse by Fred- 
holm determinant of a matrix integral operator [11] 



i^GSE(T) = Vdet(l-P(e,r7)), 



;i.74) 



and 



where 



;i.75) 



- 1 1 /"^ 

Sa{^, V) =2^Airy(e, ^) - 4 Ai(e) j Ai{t)dt, 



-J KAiryit,v)dt + -J Ai{t)dt j Ai{t)dt. 



;i.76) 
1.77) 
:i.78) 



The equivalency of (1.74) and the formula (1.31) is established in [29] and |31j . 
Now we can define Fqsei as 



i^GSEi(T) = Vciet(l-P(e,r7)), 



;i.79) 



and 



1.80) 
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where 

f4(e,^)=54(e,^) + ^Ai(o, (1.81) 

W,i^,r])=SD,{^,r]), (1.82) 

1 poo -| poo 

The matrix kernel P(^, rj) seems to be new in the hterature. However, the distri- 
bution function Fqsei is not new: 

Theorem 1.3. 

i^GSEi(T) =Fgoe(T). (1.84) 

1.5 Structure of the thesis 

In chapter [2] we reproduce Baik, Ben Arous and Peche's resuh on the hmiting distri- 
bution of the largest sample eigenvalue in the complex spiked model, by the method of 
multiple orthogonal polynomials suggested by Bleher and Kuijlaars ff\. In chapter |3] 
we use the same idea and the method of skew multiple orthogonal polynomials, to find 
the limiting distribution of the largest sample eigenvalue in the rank 1 quaternionic 
spiked model, with the help of a combinatorial result of a = 1/2 Jack polynomials 
(quaternionic Zonal polynomials) [TU] . 

To get the limiting distribution, we need technical results of asymptotic analysis. 
We put all such results involving contour integral in chapter [5} and when we prove 
theorems in chapter |2] and [3} we quote the results therein. 

The limiting distribution of the largest sample eigenvalue in the complex spiked 
model has a phase transition phenomenon, and in the rank 1 case it is an interpolation 



from -Fgue to Gaussian via Fqq^ [6J. In chapter we prove theorem 1.3, which 



together with theorem 1.2 gives the interpolation result for the rank 1 quaternionic 
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spiked model, which is from Ggse to Gaussian via Fqoe- We also give conject 
for more general phase transition phenomena. 



Chapter 2 

Complex spiked model 



In this chapter, we consider the complex Wishart ensemble unless otherwise stated. 



2.1 Determinantal joint p.d.f. formula 

For the complex Wishart ensemble, we have an advantage that the complex Zonal 
polynomial Ck(x) is the same as the Schur polynomial s^ix) up to a constant multiple. 
To be precise, we have [2^] 

C.ix) = 777^5«(x), (2.1) 
where H{k) is the product of hook lengths of n. If k = K2, . . .) and 1{k,) = I, then 

H{t^) = X{'\{{^TmS,j)+\eg,{i,j) + 1), (2.2) 

i=i j=i 

where 

axm^,{i,j)=Ki-j, (2.3) 

leg^(^,j) =min{2' \ ki' < j}. (2.4) 
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The nomenclature of arm^ and leg^ is most clearly shown by the Young diagram of 
the partition k. For example, in the Young diagram of the partition k, = (4, 3, 3, 2, 1) 



which is drawn in figure 2.1, armK(2, 1) is the number of squares to the right of the 
1-st square in the 2-nd row, and leg^(2, 1) is the number of squares below it. Usually 
we call axmi^{i,j) + leg^(i, j) + 1 = hf^{i^j) the hook length of the (i, j) square in the 
Young diagram of the partition n. 











(2,1) 






























r 





Figure 2.1: Young tableau of partition (4, 3, 3, 2, 1) 



Let us consider the rank 1 spiked model first. We assume the population 
eigenvalues are (A^ — 1) I's and the other one 1 + a. With M measurements, by 



(1.55), the joint p.d.f. of sample eigenvalues is 



1 



N 



M'=C(.)(Tt^)Qfc)(Ai,...,A;v) 



C(k){lN) 



(2.5) 



Since we require the index k, \- k of complex Zonal polynomials to satisfy /(k) < 1, 
for any k, there is only one qualified partition k, = (k). Therefore the joint p.d.f. of 
sample eigenvalues for the rank 1 spiked model is much simplified. 
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Here we notice that H{{k)) = k\, so that C(fc)(x) = S(fc)(a;). We have 
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1 + a \1 + 

{N + k-l)\ 



S{k)ilN) 



{N-l)\k\ 



(2.6) 
(2.7) 



Therefore 



P(A) 



1 

C 



N 

v{xri[x 



M-N-MX 



fc=0 



{N + k-iy. 



■An). (2.8) 



Schur polynomials have the well known determinantal representation [T^] : For any 
partition k = (ki, K2, . . . ) with 1{k,) = I, 



1 


1 




1 




Xi 






Xn 




x^- 




X2 


-1 






-1 


N- 


l + Kl 


N-l- 
X2 




N- 

Xjy 










/V-1- 
X2 


fKl 


■ ■ Xj^ 


H 


-Kl 




1 


1 




1 








Xi 


X2 




Xn 










X2 


1 









(2.9) 



where the denominator is the Vandermonde and the numerator is different from the 
Vandermonde only at the last / rows, with the power of entries of the j-th last row 
increased by kj. 
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We have 



'(A:)(Al, . . . , \n) 



1 1 

Ai A2 

\N-l+k \N-l+k 
A2 



V{X) 



1 



A 



.N~2 
\N~l+k 



(2.10) 



and 



/c=0 



(AT + fc- 1)! 



where 



1 1 


1 


Ai A2 


Xn 


Af-2 A^-2 


\N-2 


y(A) 



(2.11) 



1 + a 



-MA, 



E 

fc=0 



(Ar + A;-1)! Vl + a ' ' 



l + a 



-M 



-(7V-1) 



(2.12) 



By row operations, we can change the term e (j^MAj) in (2.11 ) into (jx^M) 



-(iV-l) 
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and we have 



1 ^ 

3=1 



1 

Ai 

,N-2 



1 

A2 

N~2 



c 



V{X) 



1 

Ai 



N~2 



-MX, _ 
1 

A2 
\N-2 



Af ^ A2 

g l + a g 1 + a ^ 



1 

Aat 



V{\) 



g l + a ^ g l + a 



1 

g l + a 



AT 



M-N -MXj 



(2.13) 



We can get similar result for the spiked model with rank r with the same idea, 



special values of Schur polynomials like (2.6) and (2.7) and more laborious work: 



Proposition 2.1. In a rank k spiked model, let r population eigenvalues be 1 + ai, 
. . . , 1 + Or and other N — r population eigenvalues be 1 . Some aj 's may not be 
distinct, but they assume values ai < ■ ■ ■ < Os with ri of them being ai, . . . , Vg of 
them being Qg one? = r. If we take M measurements, the joint p. d.f. of sample 
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eigenvalues is 



1 

Ai 



N-r-l 



A 



1 

A2 



2 



Pi 



1+a: 



MAi 



Pr 



1 

Aat 

,7V-r-l 



{y^MX,) Pr.[-^MX,) ... Pr\-^MX,) 



N 



M-N -MXj 



(2.14) 



where the determinant is similar to a Vandermonde, the only difference being in the 
last r rows: If X]i=i^ Vi < r' < X]i=i ^'^^ ~ ^l=i ~ ^' > then in the N — r + r' 
row, the entries are of the form pf ^j^^MAjj, where 

pj{x) = x^-^e"". (2.15) 



Later in this thesis we will denote the second determinant in (2.14) by V{X). We 



will not give the inductive proof of proposition 2.1 like the formula (2.13) for the 



r = 1 case, because it can be proven much easier by the Harish-Chandra-Itzykson- 
Zuber (HCIZ) formula, see e.g. [21]: 



Lemma 2.1 (HCIZ formula). Given two N x N Hermitian matrices X and Y , each 
having distinct eigenvalues Xi, . . . , Xn and yi, . . . , y^, we have 



f ,Tr(xuYU-^),rr_ 1 det(e"'i^^; 
Mn) C V{x)\ 



1<«J<JV 



x)V{y) 



(2.16) 
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Sketch of proof of proposition 2. 1 by HCIZ formula. By the HCIZ formula, we can 



simplify the joint p.d.f. (1.35) of generic Wishart ensemble (i.e., all population eigen- 
values are distinct) as 



1 



TV 



P(A) =^V{X) det(e-^'^"^Oi<.,i<^ H ^, 



M-N 
l<iJ<N I I ■^j 

i=i 

N 



^— V(A) det(e*"«^Oi<.,<TV n \' 
C{Ii,...,In) fJi 



M-N^-MXj 



(2.17) 



where we denote li = \ — , and we regard {/j} as a set of parameters, so that the 
constant C is a function of Ws. 

If (Tj's are not distinct, which is equivalent to that /j's are not distinct, then 



det(e 



AlliXj- 



i<i,j<N 



0, and heuristically, to make -P(A) a p.d.f. whose total probability 



is 1, C(Zi, . . . , In) must be also 0. In that case, formula (2.17) becomes ^, and we can 
apply I'Hopital's rule to these multiple /j's to get a reasonable formula. 

For example, if /i = • • • = In-t = and iN-r+i, ■ ■ ■ , In are distinct numbers other 
than 0, then we have 



P(A) 



r'det(e^ 



)l<i,j<N 



VC{h,...,lN) 



N 



1 

Ai 



1 

A2 



V{\)1[\ 

/jV_r— j = l 

1 



M-N -MXj 



Af-^-1 A 

gA/ijV-r + l-^l g-'W''iV-r+l 



N-r-1 
2 



,A«jvAi 



^MInX2 



VCih,...J 



A 



N-r-l 
N 



N 



N 

v{\)X[\ 



(2.18) 



M-N ~M\j 
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where T> is the differential operator 



V 



Q(N-r){N-r-l)/2 



(2.19) 



The formula (2.18) agrees with (2.14), if we regard VC{li, . . . ,In) as the constant. 
For the case that Ijy^r+i, ■ ■ ■ , In are not distinct, we apply THopital's rule also to 

□ 



them and still get (2.14). 



2.2 Multiple orthogonal Laguerre polynomials 

Our goal is to find the probability that the largest sample eigenvalue is less than a 
certain value, which is 



fT nT 

(max(A) <T)= I ... P{X)dXi . . . Ajv 



^0 



(2.20) 



To evaluate the integral of determinants (2.20), we change it into a determinant of 
integrals [S]: 

Lemma 2.2 (de Bruijn's). For any two sets of functions fi, . . . , f^ and gi, . . . , 
defined over [a,b], we have 



flixi) ... /i(xAr) 

/jv(a;i) . . . fN{xN) 



gi{xi) ... gi{xN) 



dxi . . .dxN 



gN{xi) . . . gN{xN) 

= ra! det I / fi{x)g.j{x)dx 



(2.21) 



The verification of the integral formula is straightforward. 
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Before the application of lemma 2.2, we are going to do some preparation to (2.14). 
Let (fo, . . . , fN-i be polynomials of degree 0, . . . , — 1 respectively, i.e., ^Pj{x) is 
a linear combination of functions 1, x, . . . , , which are functions defining the first 
J ' + 1 rows in the Vandermonde matrix ^(A). Similarly, let ipj [0 < j < N — 1) he a 
linear combination of the j + 1 functions which define the first j + 1 rows in V^(A), so 
that ifo, . . . , ipN~r~i are polynomials but (fN-r, • • • , 'pN-i are not. We require that 
for < i, j < - 1, 



ipi{x)ipj{x)x^-^e-''''dx = Sij. (2.22) 







These orthogonal conditions cannot determine (fj and (fj uniquely, since we can mul- 
tiply a constant C to ipj and 1/C to (pj. For 0<j<A^ — r — 1, the conditions for ipj 



and (pj are symmetric, so (pj = dpj, a constant multiple of ipj, and (2.22) gives 



oo 

M-N-Mx 



Cipi{x)ipj{x)x'''-^e-^'^dx = 6^j, (2.23) 







so that we can choose for arbitrary Cj 



^,{x) =C,Lf'-''\Mx), (2.24) 

■I ]\^M-N+l 



smce 



oo 



t-''\x)Lf'-''\x)x''^-^e--dx = 6, /^'^^^^- . (2.26) 
J- 

For notational simplicity, we denote the inner product of two functions f{x) and 

g{x) on [0, oo) by (•, ■)2: 

{f{x),g{x))2= I f{x)g{x)x^'-^e-^'^dx. (2.27) 
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For j > N — r, we index cpj and (pj by a triple {r',s',t'), which appears in the 



statement of proposition 2.1, and is defined as 



r' e {l,...,r}, r'=[^rjj+t', 1 < t' < r. 



(2.28) 



It is Bleher and Kuijlaars' observation that [7] 



Proposition 2.2. Given {r',s',t') defined as (2.28), we have 



X 



^Mxz 



2-Ki 



{z-l) 



^M-r+r' 



n 



1 + a, 



1 + a„ 



dz, (2.29) 



and 



(pN-r+r'-l[X) 



Me 



Mx 



-Mx 



(1 + asi)Cr'2ni 



{z-\ 



1 + a. 



1 + a^i 



-t' 



dz, (2.30) 



where Or' is an arbitrary constant, H is a contour around z = and is to the left 
of the points z = 1 and z = (j = 1, . . . , s'), and T is a contour containing the 
points z = 1 and z = (j = 1, . . . , s'), and is to the right of z = 0. 

Sketch of proof. By the residue theorem, we can check that ipN-r+r'-i{x) defined in 
(2.29) is a polynomial of degree N — r + r' — 1, and (fN-r+r'~i{x) defined in (2.30) is 



also in the correct form: For any r', 



ipN-r+r' -lix) = CqI + CiX + ■ ■ ■ + C^-r-lX 

ai 



N-r-l 



+ CN-rPl 



l + ai 



Mx H h CN-r+r'-lPt' 



as' 

1 + Oo 



Mx , (2.31) 
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where the j-th term is a constant cj times the function that defines the j-th row in 



the determinant in (2.14). Then we can check that for A; = 0,...,A^ — r — 1, 



{ipN-r+r'-l{x),x'')2 

2m t z^^-+-' 



'0 

Cr' 



n 



k„Mx{z-l)^^ 



X e 



(z-l) 



N-r / ]^ 



iz-1 



^M-r+r' 



+ a, 



1 + a„/ 



z 



dzdx 



2Tii Js (M(l - z 

k\Cr' 



M-r+r' \ n (^"r^ ' ' ' ^ 



l + as 

t'-i 



1 + 



dz 



N-r-k- 



1 \ * ^ 



1 + ^M-r+r' ' 

(2.32) 



which is by the residue theorem. Similarly, we can prove for j < s' — 1, z < r,- or 



j = s', i < t', using (2.15), that 



(pN-r+r'-l{x),Pi 



1 + aj 



-Mx 



(2.33) 



so that by linearity, for < j < N — r + r' — 1, using (2.31 ), we have 



{ipN-r+r'-l{x),ipj{x))2 = 0, 



(2.34) 



In the same way, we can check that for < j < N — r + r' — 1, 



{x\(pN~r+r'~l{x))2 =0, 
{iPj{x),(fN-r+r'~lix))2 =0. 



(2.35) 
(2.36) 



We can compute the leading term of ipN^r+r'-ii^), i.e., the x^ ^ term, from 
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(\2.29L which is 



Cr' 



{Mxz 



M-r+r'-l 



{z-l) 



N-r 



x^-^27Ti Js (M - r + r' - 1)! z^'^-'-+^' 



-1) 



N~r+r'- 



1 + as 



t'-i 



dz 



Cj-' 



n 



Af-r+r'-l 



. (2.37) 



Therefore by (2.35) and (2.37), if we denote Cr' = (11^=1 



rs'-l 1 



{(fN-r+r'-lix), 'fN~r+r'-lix))2 

_^-^N~r+r'-l 
1 



(M-r + r'-l)! ' V^^-r+r'-Ux, 



27ri 



{M -r + r' -l)\l + as' Jo 

M-r+r'-l 

n 



X 



M-r+r'-l 



{Z~l) 



N-r 



1 + 0.^ 



1 + 



dzdx 



i\/fM-r+r' ry i 



(M - r + r' - 1)! 2m Jr Vio 

(z - 1)^-^'- 1 n 



X 



M-r+r'-l -Mxz 



dx 



1 + 



1 + 



dz 



-1) 



-1) 



jV-r+r'-l 



Cr' - 



27Ti Jy z{z — 1) 



N-r 



z — 



Cr' - 



27Ti 



1 



„ z{z - 1)^-^ 



p/ z{z — 1) 

s'-l 

z 



N- 



^s'-l 

n 



1 



1 + % 
1 



1 + 



dz 



n 



1 + 



1 + a, 



1 + a^' 



1 + a« 



dz 



(2.38) 



where F' is a contour including 0, 1 and (?' = 1, . . . ,s), and T" is a contour 
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including and excluding 1 and We can deform T' to infinity to see that 



— / 1 

2m Ir, ziz-l)^--' 



n -r 



+ a, 



'1 \ / 1 
z — 



1 + a„ 



-t' 



dz = 0, (2.39) 



and by the residue theorem see that 



1 r 1 

%u ziz- 1)^-^ 



n -r 



+ a,- 



1 + a..' 



dz 



Cr' 



and we check that {ipN-r+r'-i{,x),ipN-r+r'-i{,x))2 = 1. 



(2.40) 

□ 



Although the integral formulas for (pN^r+r'-i{x) and ipN-r+r'~i{x) seems strange, 
if we compare ipN-r+r'-ii^) with the integral representation of Laguerre polynomials 



j^{M-N) 



(Mx) 



-.Mx 



-Mxz 



Z 



M-N+j 



z - IVTT^-' 



2vri jp 

and ipN-r+r'-i{x) with another integral representation of Laguerre polynomials 



(2.41) 



- (M-N) 



(Mx) 



(M-N + j)\ 1 



(2.42) 



and find that they are variations of Laguerre polynomials. The (pN^r+r'~i{x)^s are 
called multiple Laguerre polynomials of type II, and the (pN-r+r'-ii^Ys are equivalent 
to — although they are not polynomials — multiple Laguerre polynomials of type I, by 
Bleher and Kuijlaars [7]. 
We know that 

1 



V{X) 



C 



V^Af-i(Ai) 



V5o(A7v) 



(2.43) 
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and 



V{X) 



c 



SO that we can write the formula (2.14) for the joint p.d.f. as 



P(A) 



C 



V5o(Ai 



V5o(A 



N 



V5o(Ai 



V5o(A 



N 



V9Ar-i(Ai) ... (Pn-i{^n) 



N 



M- 
3 



and by formulas (2.20) and (2.21), we have 



max(A) < T) = -det ( / ^i{x)^j{x)x'"'~'' e-^'^'dx 



0<i,j<N-l 



where we choose the fi and gj in (2.21) to be 



M-N M, 



fi{x) =ipi_i{x)x 2 6 2"^, 

gj{x) =ipj^i{x)x 2 6 2. 



By (2.22), we have 



T poo 

v/t 



and so 



det( / ipi{x)^j{x)x^-^e-^''dx 



0<«,j<Af-l 
oo 



det I /tv - ( / ^i{x)^j{x)x^-^e-^''dx 



T 



0<i,j<N-l 
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To evaluate the determinant in (2.50), we apply the formula 



det Jtv - / fi{x)gj{x)dx 



det I / 



XiaMi^) (^Jjix)9j{y)^ xia,b]iy)j > (2-51) 



where X[a,b] is the indicator function for the interval [a, b], and the latter determinant 
is the Fredholm determinant for integral operator. Since the integral kernel on the 



left hand side of (2.51) is of finite rank, we can check the identity directly. Consider 



the parametrized determinant 



det ilN-t (^j fi{x)gj{x)dx 



1 + Cit + C2r + ■■■ + c^r, (2.52) 



i<«,i<JV, 



we can compute the coefficients degree by degree {K{x,y) = J2f=i fji^)9jiy))- 



xi, xi)dxi 



ci = - ^ / fj{x)gj{x)dx = - K{ 

j = l 

^ / pb pb pb 

C2 = ^ ( / f\{x)gi{x)dx I fj{x)gj{x)dx - / /, 



(2.53) 



x)gj{x)dx / fj{x)gi{x)dx 



1 
^2! 



b fb 



K{xi,xi) K{xi,X2) 

K{x2,Xi) K{X2,X2) 



dxidx2, 



(2.54) 



and we find the right-hand side of (2.52) is the same as the right-hand side of (1.57) 
for t = 1. 

Now using (2.46), (2.50) and (2.51) we get the Fredholm determinantal formula 
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for the probability that the largest eigenvalue is less than T: means X[T,oo)(a^)) 

P(max(A) < T) = 

1 
C 



^ det I / - ( J2 '^j{x)<pj{y)x'"^'' y"'^'' e '^'(^+^) ) x{x) ) . (2.55) 



Furthermore, we can determine the constant C = 1, since letting T oo, P(max(A) < 
T) — i> 1 and the Fredholm determinant on the right hand side also approaches 1. To 
undertake real calculations, we need methods to evaluate of the Fredholm determinant 



other than the formula (1.57). First, we have the result on taking the limit, see, e.g. 

m- 

Proposition 2.3. // a series of integral operators Kn approaches K in trace norm, 
then 

lim det(/ - Kn) = det(/ - K). (2.56) 

n— >oo 

Since as M — oo, we expect that the fluctuation scale of the largest eigenvalue 
shrinks, depending on M. We will consider for any M the probability P(max(A) < 
p + qT), where p and q may depend on M. (It turns out that p is a constant.) If we 
denote 

N-r-l 

K2a{x,y)= J2 Vj{x)^j{y)x^y^e-f^''+y\ (2.57) 

j=0 
N-l 

K2b{x,y)= Yl Vj{x)vAy)x'^y'^e-f'-^+y\ (2.58) 

j=N-r 

K2ix,y) =K2aix,y) + K2bix,y), (2.59) 



and 



K,i^,r]) =qK,ip + q^,p + qv), (2.60) 
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where * stands for 2, 2a or 26, then we have the determinant al formula 



(max(A) <p + qT) =det (/ - X\p+qT,^)ix)K2{x,y)x[p+qT,oo)iy)) 
= det (I-X{0K2{^,V)X{V) 



{2.61] 



K2a{x,y), which is composed of Laguerre polynomials, is the kernel for the LUE, 
and we can write it in an integral form: 

Theorem 2.1. 



K2aix,y) = -M 



M-N M 

X 2 e 2^ Jo 



27Ti 



-M(y+t)z_ 



z-l 



\N-r 



dz 



2ni 



N~r 



W 



M-r 



dw dt. (2.62) 



Proof. Because of ([2^, ([2^, ( p^ , ( |2l^ and ( p^ , we have the integral rep- 
resentation 



N-r-1 



j=0 



(M- + j)! 



M y 



Af-iv Ai,, N—r~l 



2 e 2 



(b dz (b dwe 



-Myz 



Z 



M-N+j 



^Mxw 



{w - ly 



[z - 1)^+1 



w 



M-N+j+1 



(2.63) 
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We can write the sum of integrands in (2.63) as 



N-r-l 

E 

j=0 



Mxz 



Z 



M-N+j 



[z - 



_^Mxz ^—Myw 



-Myu, {w-iy 



/ z{w - 1) 

j=0 



_^Mxz ^—Myw _ 



z{w—l) 
{z—l)w 



N-r 



W 



M-N 



{z -l)w 1 _ ^("'-^) 



{z—l)w 



^Mxz^M-N^-Myw ^ 



Z — W 



M-N 



W 



Mxz 



Z — W 



-Myw 



[w-l 



{z-l) 



N-r 



W 



M-r 



(2.64) 



By residue theorem, since T and E are disjoint, and for the variable z, the pole z = w 
is outside of F, 



z — w 



M-N 



0. 



w 



(2.65) 



On the other side, since K(w — z) is always less than 0, 



z — w 



(2.66) 



so that we have 



M 



dz (h dw- 



-Myz 



Z 



M-r 



hlxw 



[w-l) 



N-r 



z-1)^-'- 

M-r 



W 



M-r 



' dz i dw --''^y+'> ' 



^M{x+t)w 



[w-l 



^N-r 



2m 



^M{y+t)z 



{z - lY-' 

^M-r W 1 

(^-1)^-'^ J V2^.E 



w 



M-r 



\N- 



W 



M-r 



(2.67) 



Put ( 2.63 )-( 2.67) together, we get the result. 



□ 
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Now K2a{x,y) {K2a{C,,v)) is expressed by two functions and K2b{x,y) {K2b{C,,ri)) 
is a finite rank operator. To undertake the asymptotic analysis, we need two propo- 
sitions on the convergence in trace norm: 

Proposition 2.4. Let {fj,n} o,nd {gj.n} A — J — ""^^ 1 < n < oo) he 2m series of 
functions on [T,oo) and fj^n — * fj, gj,n gj in norm on [T, oo). We have the 
convergence in trace norm of operators (Kn{x,y) = "Y^jLi fj,n{x)gj,n{x) , K{x,y) = 
T.7=ifj{x)gj{x)): 

hm x{.x)Kn{x, y)xiy) = xix)K{x, y)xiy)- (2.68) 

Proposition 2.5. Let fn{x), gn{y) be two series of functions on [T,oo) and in L^ 
norm we have 

hm ||(/„(x) - f{x)){x - T)\\l2^[t,oo)) = 0, (2.69) 

n— +00 

hm \\{g^(y)-g(y))(y-T)\\L2^[T,oo)) = 0, (2.70) 
n^oo 

then we have the convergence in trace norm of operators (Kn{x,y) = fn{x + 
t)gn{y + t)dt, K{x, y) = f{x + t)g{y + t)dt): 

hm xix)Kr,{x, y)xiy) = xix)K{x, y)xiy)- (2.71) 



We can verify proposition 2.4 by the definition of trace norm. Proposition 2.5 
is essentiahy a fact of trace norm convergence [17]: If J„, Jn are Hilbert-Schmidt 
operators and In I, Jn —>■ J in the Hilbert-Schmidt norm, then the product InJn 
and IJ are trace class operators and InJn I J in trace norm. Her we take J„ as the 
integral operator from L^(]R) to L^(]R) with the kernel x{x) fn{,x + y)x[fd,oo){y) , and J„ 
also an integral operator from L^(]R) to L^(]R) with the kernel X[Q,oo){.x)gn{x + y)x{y) ■ 
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For an integral operator, the Hilbert-Schmidt norm is equivalent to the L^ norm of 
its kernel as a 2 variable function. In our special case, the convergence results of 



L^(]R^) is equivalent to the convergence of L^{[T, oo)) in (2.69) and (2.70), due to the 
Fubini's theorem. 



To apply propositions 2.4 and 2.5, we sometimes need to conjugate the integral 
operator by a weight function: 



Proposition 2.6. We have 



det(l - K{x,y)) = det(l - fix)Kix,y)f-\y)) 



(2.72) 



for any function f which make the right hand side of (2.72) well defined 



Proof. A direct application of the definition of Fredholm determinant (1.57). □ 



1-7 



If we take f{x) = x^^^-^^/^e^^) , then by ( [2^ , and ( [2^ ( |2^ we 

have 



r"00 

f{x)KUx,y)f-\y)=-M^ I e-^^^^^+'^ilj{x + t)e^^^"^y+^iy + t)dt, (2.73) 
f{x)K2,{x,y)f-\y) =M V -^e-^*^(^+*Vr'(x)e^^'^(^+*)^,,(l/), (2.74) 



r'=l 
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where 



1 ^ Mxzi^ 1)^ 



27ri 



-Myz_ 



Z 



M- 



2m '■'""(z-l)^-'-'^'^' 



(2.75) 
(2.76) 



V^r'(2/) 



27ri 



-Mxz 



yM-r+r'-l 



'_i A'-i 



1 + 



(2.77) 



n 



1 \--. 



1 + a,' 



-t' 



dz. 



(2.78) 



2.3 Proof of theorem 11.1 



We give proofs of all the three parts separately. In this section, we let x = p + and 
y = q + qri. 



2.3.1 The -1 < as < 7"^ part 

In this case, we choose p = (1 + 7~"^)^ and q 



7M2/3 • 



and by (2.60), write (2.73) as 



'0 



(2.79) 



where 



vl/(^) j2±}^M^/\-lf 



1 



M 



7 

(7 + 1)'/' 
7 



(7 + I) 



M-N 



^(7 + 1)*'-^ ^ 



7 



(2.80) 
(2.81) 
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Since (5.51) and (5.71) give the L convergence results 



lim mio-i-iYmm-ny 



lim 



([T,(»)) - 0' 



(M/(r^)-(-irV^'Ai(r^))(r^-T) 



L2([T,cxd)) 



(2.82) 
(2.83) 



we get the convergence in trace norm by proposition 2.5 



Jo 

-XmMry{^,V)x{v)- 



(2.84) 



Similarly, for any r' = 1, . . . , r, by (2.74) and (2.60) 



7 



(7 + 1)4/3MV3 1 + 



^-^ 1 + a.i 



(2.85) 



where 



e ^+i*'^^^r'(P + ?0> 



(7 + 1)'^-^ 
(7 + 1)"^-^ 



7 



7^ 



(2.86) 
(2.87) 



Because (5.50) and (5.70) imply the L convergence results {Cr> is defined in (5.45)) 



hm ^r'{i)x{i) =(-l)V-^ (7 + l)'^a-iAi(e)x(0, (2.88) 

hm ^r>{n)x{n) = ^Kv)xiv), (2.89) 

M^oo 7*^ (7 + 1) Cr' 

(2.90) 



we get by proposition 



2.4 



the boundedness in trace norm [C is a large enough positive 
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constant) 



<c, 



Tr 



SO that in trace norm 



(2.91) 



lim xiOf{x)K2k{^,v)r\y)xiv) = 0. 

M^oo 



(2.92) 



Therefore, 



lim det 1 - xiO^i^, v)x{v) 

M^oo \ 

hm det (l - x{Ofi^)Ki^,v)xiv)r\y) 
lim det (l - x{i)f{x)K,a{i.v)r\y)x{v) 

det(l-x(O^Airy(e,^?)x(^))- 



(2.93) 



2.3.2 The = 7^ part 



In this case we still choose the same p and q as in the previous case, but we need to 
conjugate the kernel not only by /(x), but also e^/^. If we consider 



f{x)e^'^K2a{S,, V)r\y)e~"^^ = - e(«+*)/=^^(e + t)e~'^"+'^/^^{r] + t)dt, (2.94) 

Jo 



and similar to (2.82) and (2.83), (5.51) and (5.71) imply also the convergence 
results 



^lim \\{e^/^m - i-lYe^^'Amm - ^)|L.([T,oo)) = 0' 



lim 

M— >oo 



(6-^/3^(7^) - (-1)^-17-^6-''/=^ Mv))iV - T) 



(2.95) 
(2.96) 



and we can get the result of trace norm convergence similar to (2.84) 



hm x(e)/(a;)e«/=^i^2a(e,r])/-'(2/)e"''/\(r^) = x(Oe«/'i^Airy(e,r/)e-''/\(r/). (2.97) 



M^oo 
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we can still get by (5.51), (5.71) and proposition 2.5 the trace norm convergence 



POO 

--xiO / e(«+*)/=^ Ai(e + t)e~(''+*)/2 Ai{7] + t)dtxiv) 
Jo 



(2.98) 



We can write the formula (2.74) as 



+ V -^^e^'^^eyiOe-'^'^^fAv), (2.99) 
1 + 7 ^ 

r'=r— r, + l 



where 



(7 + 1)^ 

7 

(7 + 1)^ 

7 



M^/M (-1 



.iv 7 



e-^'^'"^,/(p + gO, (2.100) 



Ml/3 



(7 + 1) 
(7 + 1) 



M-N 



7 



i^^^yipr'ip + qv)- (2.101) 



In the same way of (2.92), we have the convergence result in trace norm 



lim 

Af^oo 



7 



(7 + 1)4/3M 



^r'(^) I 



vr'=l 



(2.102) 

However, if r' = + t' , t' = 1, . . . , r^, (5.86) and (5.98) yield the convergence 

in norm 



lim e^/'^iOt'MO =(-l)V-'-'(7 + lY'Cr'-t'hlY'-'e^^''t'^''\0, (2.103) 

(-ir(i+7-^) 



M^oo 

lim e~'^^^'^{r])t'yxiil) 



'Y-'-'ii + iycr'-t' 



l)*'-ie-V3^(0(^)^ 



(2.104) 
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so that by proposition 2^ we have the convergence in trace norm 

M— »oo I ^ — ^ 1 + 7 ^ / 

\r'=r-rs+l / 

e«/3 (^^t^'\Os^'\v?j e"''/', (2.105) 

which is exactly the trace norm hmit of x{0fi^)^^^^^2bi^,v)f~^iy)^~^^^x{v)- 
Therefore, 



hm 


det 




M— »oo 




= hm 


det 











= det 






= det 







,lim det fl - x{0f{x)e^^'{K2a{^,v) + K2,{^,v))x{v)r\y)e-''^\ 
det ( l-x(e)e«/3 |^irAi.y(e,r^) + XJt(^)(03(^H^)j e-''/=^x(r/) j ^^.lOG) 

2.3.3 The ag = a > 7^ part 

In this we choose p = {1 + a) (l + and g = (1 + a) ,^JV-^^^^j^, and conjugate 
the kernel by /a(x)e2«/3, where /^(x) = x^^'^'^^/'^e^^)^'' , Then by ([Z62|, (|2J5| and 



(2.76) we have 



Jo 

(2.107) 
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where 



M 



e 1+" 



72a2 J (-a)^(l + a)^^-^ 



^'^(r^) =(1 + a) W 1 - — M(-a)^(l + a)''~^ e^^^^Pip + qv), 



(5.134) and (5.155) yield that (C is a large enough positive number) 



hm \\e'^/'<i^^im-T)\\LH[T,oo))<C, 

'l—rCO 

hm ||e-2''/3^'^(r/)(r/-T)|U.([T,oo)) = 0, 



M^oo 



then by proposition 2.5, we have in trace norm 



hm x(e)/a(x)e2«/3if2a(e,r/)/-^(l/)e-2''/\(r^) = 0. 



M^oo 



For the K2b part, we have the formula similar to (2.99) 



1 



■2r)/3 



j=r-rs+l 
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where 



^Me)=(l + a)W 1 



M 



e 1+" 



72a2 J (-a)^(l + a)*^-^ 



(2.114) 



^J!,(r/)=(l + a)W 1 



M(-a)^(l + a)^^-^eiT^^V^,,(p + gr/), (2.115) 



M 



-a)^(l + a) 



M-N 



A'ip + qO, (2.116) 



7^0^ 



^t'yiv) = (1 + a). / 1 - — M (-a)^(l + a)^^-^ei^^^.,(p + qv). 



(2.117) 



(5.133) and (5.155) yield that for r' = 1, . . . , r — r^, (C is a large enough positive 



number) 



hm \\e'^/'^^AOxmLH[T,oo))<C, 

d — >oo 

hm ||e-2''/3§M^)x(^)llL^([T,oo)) = 0, 



M— >oo 



(2.118) 
(2.119) 



so that similar to (2.102), we have the trace norm convergence 



lim 

M^oo 



(1 + «)a/(1-:^ 



I X;e2«/=^vl/,,(e)e-V3^^,(^) ] ^(^) 



M 



vr'=l 



0. 



Tr 



However, (5.132) and (5.154) give the L norm convergence results 



(2.120) 



hm e^«/3v^^,,,(0x(0 =^^^' 
hm e-^'^'^ny{v)x{v) 



2n 



e 2 



t'-i 



{l + aY'-^Cr'-f {f -ly. 



(2.121) 
(2.122) 
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Then by proposition 2.4, we have the trace norm convergence result 



\j=r-rs+l / 



xiv) 



xiOe'^/' f E ]j71^^^-(^)^^-^^)""^ I ^'"'^'^(^)- (2.123) 



Therefore, 



lim det(l-x(0^(e,^)x(^7)) 

M— >oo 

hm det (l - xiOfa{x)e'^^'{K2a{^,v) + K,,{i,r,))f-\y)e~'^^l' 



where in the last step we conjugate the kernel by e 3 + 4 



Chapter 3 

Rank 1 quaternionic spiked model 



In this chapter, we consider the rank 1 quaternionic Wishart ensemble unless other- 
wise stated. Since we have only one parameter ai in the p.d.f. of sample eigenvalues 



(1.52), we denote it as a. 

The reader should be cautious that some notations, for instance, ipj and ipj, are 
defined differently from their definitions in chapter [2| and V^^(A) is not the fourth 
power of V^(A). 

3.1 The joint distribution function 

In this section, we prove 

Theorem 3.1. The joint probability distribution function of A in the quaternionic 
spiked model is 

P(A) = ^V\\) n (Af ^^-^)+^e-2*^^^) . (3.1) 
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Here 



1 

Ai 
A? 




1 

2Ai 



^2N-2 - 2)Af 

2A/Ai a o /\^^Tx:72AfAi 



ei+a 



l+a 



Atv 




1 

2AAr 



i2iV-2 



ei+a 



^ (2iV-2)A; 

l+a 



( 



the determinant of a 2A^ x 2A^ matrix whose (2A^, 2k — 1) entry is ei^^*^'^*, (j, 2A; 
entry is A'^"^ for j = 1, . . . , 2N — 1, and 2i-th column is the derivative of the {2i — 1 
column. V"'^(A) is a variation of the V{Xy appearing in the LSE (see j21j). 



Since rank r = 1, we can simplify (1.52) as 



1 



N 



P(A) = -V{Xy Y[ ^ (2M)J' <5(j)(if^)<5(i)(Ai, ■■■An) 



j=0 



Q{j){lN) 



just like formula (2.5) for the complex case. 
We have |26] 



+ a' 



l + a 



and since the number of variables is N l26| 
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so we get 



^ {2MyQu){^JQu){\i,...,XN] 



In [2n] there is an identity 



TV 



-£{3 + l)Qo)(Ai, . . . , A^)t^- = n TTtV^- (3-7) 
Comparing it with the well-known identity for Schur polynomials 

oo ^1 

5^s(,)(Ai,...,A;vy = nY-y^, (3.8) 

we get the identity 

(i + l)Q(j)(Ai, • • • , Aat) = S(j)(Ai, Ai, A2, A2, . . . , Aat, Aat), (3.9) 

with each Aj appearing twice as variables of the sq). For notational simplicity, we 



denote the right hand side of (3.9) as S(j)(A), which is a plethysm 



^(j)(Ai, . . . , Xn) = o 2pi(Ai, . . . , Aat). (3.10) 



Now we get 



j=0 



(2M)^Q(i)(Tf^)Q(i)(Ai,...,A^) 



N) 



E 



,=0 nU(2iv+.) 



1 + a 



2M %)(Ai,...,Ajv). (3.11) 



CHAPTER 3. RANK 1 QUATERNIONIC SPIKED MODEL 



56 



Then we need a lemma to simplify (3.11) further. 
Lemma 3.1. 



%)(A) 



1 





1 





Ai 


1 




1 


A? 


2Ai 




SAat 



Af 
A? 



2N-2 



2N+j-l 



{2N - 2)A2^-3 
(2iV + j-l)Af+^'-' 



\2Ar-2 



{2N - 2)A2^-3 



(2iV + j-l)A 



2N+j-2 
N 



v{xy 



(3.12) 



w^/i the {k, 2j — 1) entry of the matrix being a power of Xj with the exponent k — 1 if 
k 7^ 2N and 2N + j — 1 if k = 2N , and the {k, 2j) entry being the derivative of the 
{k,2j — 1) entry with respect to Xj. 



To prove this lemma, we need the well known fact (see proven by L'Hopital's 



rule 



V^(A)^ 



1 

Ai 

, 2Ar-l 





1 



1 

Xn 




1 



Af-^ (2iV-l)A^ 



2Af-2 



A 



N {2N-1)X 



N 



(3.13) 



with the {k,2j - 1) entry being A^ ^ and the {k,2j) entry {k - 1)A^ ^. 

Proof of the lemma. Applying the L'Hopital's rule repeatedly with respect to X2i 
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i = 1, . . . , N , we get the identity 



1 

Ai 
A? 




1 

2Ai 



^2N-2 (2iV-2)Ar-=^ 
Af+^-^ (2iV + j-l)Af+^-^ 



An 



,2N-2 





1 

2A7V 



2N~3 
N 



{2N -2)X 
Af+^-^ (2iV + j-l)Af+^- 



1 
Ai 1 

A2a^-i (2iV-l)A2^-2 



8x28x4. ..dx2N 



X 



1 

1 



1 

0:2 



■^2 



2Af+j-l 2Ar+i-l 



8x28x4. ..8X2N 



1 



1 

2^2 



„2Af-2 



„2Ar-2 
^2 



X 



1 

Aat 




1 



Af-i (2iV-l)A 
1 1 



2N-2 

N 



2N-2 



2N-2 



^2N-l ^2N 
2N+j-l 2N+j-l 
X2N-I ■^2N 



1 

X2N-I 
„2N-2 



1 

X2N 
2N-2 



1 • • • 2N—1 IN 



JiN+j-l 2N+j-l 
Xo^/■ 1 



'2N-1 ■^2N 

=S(j)(Ai, Ai, A2, A2, . . . , Aat, Aat) = S(j)(Ai, . . . , Aat) 



X2i-l=X2i=Xi 

1=1. ...,N 



(3.14) 



from the matrix representation of Schur polynomials, and now use (3.13) to get the 
compact formula (3.12). □ 
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Substituting (|3.12l) into (|3.11l), we get 



(2M)^gO)(lf^)gO)(Ai,...,A^) 



i=o 



Q{j){lN) 



1 

Ai 
A? 

2N-2 





1 

2Ai 



A^ 
p(Ai 



(2A^-2)Af 
p'(Ai 



2Af-3 



1 



1 

Ai 
A? 




1 

2Ai 





1 

SAat 



A 



(2iV-2)A 

p{\n) P'(AAr) 



A^ 



2N-2 



{2N-2)Xi 



2N-3 



1+a 



1 



V^^(A), 



1 

Ajv 



\2N-2 
e 1+° 




1 

2A7V 



2Af-3 



(2iV-2)A^' 

l+a 



(3.15) 



where 



00 



{2N-1)\ 



2M 



X 



2N+j-l 



2N-2 



ei+. 



-2Mx 



^ j! (1 



+ a 



-2Mx 



(3.16) 



and if 7^ 2A^, the (fc, 2j — 1) entries in both matrices are A*^ ^, and the {k, 2j) entries 



are {k — l)Xj , and the 2N, 2i — 1 entry in the former (latter) matrix is p(Aj) (resp. 
eTf^2A^A,^| g^^^ ^j^g 2Ar,2i entry p'(Ai) (resp. j^2Me^^^^'). 



Proof of the theorem. Formulas (3.3) and (3.15) together give the result (3.1). □ 
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With the formula (3.1) ready to use, we get the hmiting distribution formula for the 



largest sample eigenvalue, in the same spirit as the solution of the quaternionic white 
Wishart ensemble. Our process below is closely parallel to that in [30] . 

First, we find a skew orthogonal basis {(y9o(x), (fi{x), . . . , '^2N-i{x)} of the linear 
space spanned by {1 

2N "^^Qi+a ^^}.We require that the '■Pj{x) is a linear 
combination of {1, x, x^, . . . , x^} if j < 2N — 1, while (p2N-i{x) can be arbitrary, with 
the skew inner products among them 



rj/2 if j is even and k = j + 1, 
— if k is even and j = k + 1, 
otherwise. 



(3.17) 



Remark 3.1. Due to the shortage of notations, we abuse the language so that we use 
ip and ijj in this chapter to mean functions different from those in chapter [2j 

Then we can reformulate the distribution function of A as 



P(A) 



C 



V^o(Ai) 



/o(Ai) 



1 



N 

n (Af '-^^+^e-2*^^. 

^o(Ai) ^^(Ai) 
^i(Ai) ^'i(Ai) 



f2N-l 'P2N~l 



^o(Aiv) 

^l(AAr) 



^2Ar-l(Al) V^2iV-l(Al) ••• ^27V-l(AAr) ^2JV-l(A7v) 



(3.18) 



CHAPTER 3. RANK 1 QUATERNIONIC SPIKED MODEL 



60 



where 



(3.19) 



For an arbitrary function f{x) on [0, oo), by the formula of de Bruijn 



Jo Jo 



^i(Ai) 



^o(Ai) 
^UAi) 



^o(AAr) 



^o(Aiv) 
^'i(A^) 



V'2Af-l(Al) V^2iV-l(Al) ••• V'27V-l(Aiv) V'2iV-l(Ajv) 



N 



+ /(A,))rfA, = CPf(P(l + /)), (3.20) 



j=i 



where P(l + /) is a 2A^ x 2N matrix, whose entries depend on 1 + / in the following 

way 



(P(l + /)),,fc = (7/;,_i(x)V^Li(x)-^;_i(x)^fe_i(x))(l + /(x))rfx. (3.21) 



Now we define a matrix Z as 



with 



/ ro 
-r-o 



ri 
-ri 



Tat-i 

-r7v-i y 

if is even and j = k — 1, 
—rj/2~i if j is even and k = j — 1, 
otherwise, 



(3.22) 



(3.23) 
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and define for j = 0, . . . , — 1, 77 = Z'^ip, i.e., 

V2j{x) = and r]2j+i{x) = . (3.24) 

So we liave 

+ {ijj^i{x)ij'k_i{x) - ij'j_^{x)iJk-i{x))f{x)dx (3.25) 
Jo 

POO 

=Zj,k+ / {iJj-i{x)ij'f,_-^{x) ~ij'j_^{x)iJk-i{x))f{x)dx. 



And if we denote Q(l + /) = Z-^P{1 + /), then 

iVj~iix)ip'k^iix) - ri'j_^{x)ipk~iix))f{x)dx. (3.26) 



If we choose / to be — X(t,oo), then the integral on the left hand side of (3.20), 
after multiplying a constant, is the probability of all Aj's smaller than T. In latter 
part of the paper, we abbreviate X(t,oo) to x as before. So we get for a T-independent 
constant 

P(max(Ai) <T) = CPf(P(l - x)), (3.27) 

and 

(P(max(Ai) < T))2 = det(P(l - x)) = det(Q(l - x))- (3.28) 



Now we apply a matrix version of (2.51). In linear algebra, we have the determi- 
nant identity 

det{I - AB) = det{I - BA), (3.29) 

for A an linear map from to M"* and B an linear map from to M"*, and the 
identity still holds in infinite dimensional settings Letting det mean a Fredholm 
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determinant for a matrix integral operator defined in (1.71), we describe a setting 
due to Tracy- Widom 



If A is an operator from L^(]R) x L^(]R) to the vector space with 



A 



9{x) 
h{x) 



Jo 



(3.30) 



and B is an operator from M.^^ to L^(]R) x L^(]R) with 

\C2nJ 



B 



(3.31) 



then 



and 



I-AB = Q{l-x)^ 



I - BA = I -x{x) x(2/), 

ISi{x,y) Si{y,x) 



(3.32) 



(3.33) 



where S^{x,y), IS^{x,y) and SDi{x,y) are integral operators whose kernels are 



2N-1 
j=0 


x)r]j{y) = 


Yl -(-^2i(a;)V^2i+l(y) + ^2i+l 

i=o ^ 


[x)iJ2j{y)), 


2Af-l 
j=0 




= 5Z -(^2,(a^)V^2i+l(l/) - V^2i+1 


[xW2,{y)). 


2Af-l 
1— n 


x)r]j{y) = 


—{-i^2j{x)ij2j+iiy) + i^2j+i 


[x)ip2j{y)), 


^ -iljj^xy^iy) 

j=0 


= -(^2i(a;)V'2j+i(2/) - ^2i+i 


[x)ij2j{y)). 



Remark 3.2. It is clear that the nomenclature of SDi{x,y) is due to the fact that 
SD4{x,y) is the negative of the derivative of Si{x,y). But IS4{x,y), which gets its 
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name in the same way in earlier literature in GSE (e.g., [29]), in our problem may 
not satisfy the equation 

/■oo 

IS,{x,y) = - / S,{t,y)dt, (3.38) 

J X 

since the integral on the right hand side may diverge. 
In conclusion, 

(P(max(A.) < T))^ = CMet ( I - x{x) ( ^^f^A x{y)] , (3.39) 

and we can find that = 1 by taking the limit T — > oo. We define a 2 x 2 matrix 
kernel as 

p / N ^ ^ ( Siix,y) SDi{x,y)\ 
PT{x,y) =X{x) K xiy) 
\IS4{x,y) S4{y,x) I 

^ ^ ^ (3.40) 

x{x)S4{x,y)x{y) x{x)DSi{x,y)x{y) 

^x{,x)ISi{x, y)xiy) xix)S4{y, x)xiy) 
then we have 

(max(Ai) < T)f = det(/ - Prix, y)). (3.41) 



3.3 S/^{x^ y) in terms of Laguerre polynomials 

In manipulation of skew orthogonal polynomials, we take the approach of [2|, and all 
properties of Laguerre polynomials are from [27] . 

Since Laguerre polynomials by definition satisfy the orthogonal property 

r Lf (*^-^»Lf ^^-^»x2(*^-^)e-^rfx = + (3.42) 
Jo 3- 
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and they have the differential identity 



X = n42(^-^))(a;) - {n + 2(M - N))L^^^f-''^\x), (3.43) 



it is easy to get that 



L 



(2(M-^))(2Mx),Lf^^-^))(2Mx; 



^(2{M~N)) 



d 



(2(M-N)) 



{2Mx) 



(2A/) 



]_^^2(M-Af)+l (j+2(M-JV))! 

-1)! 

2(A/-. 
(fc-1)! 



/ j_N 2(M-Af)+1 (fc+2(A/-jV))! . 
I 2A/ J 



if J = + 1, 

if A; = J + 1, 
otherwise. 



(3.44) 



So we can choose for j = 0, . . . , — 2, 



<^2j+i(a;) 



fc=0 \i 



n 



2i - 1 



»■ 2i + 2(M - A^) 



(2(Af-JV)) 
2k 



(2Mx) 



(2(M-Af)) 
2j+l 



(2Mx) 



(3.45) 
(3.46) 



and 



1 \ (2j + 2(M - AT) + 1)! 2A; - 1 



2M 



(2j)! 



n 



L2A; + 2(M- AT)' 



(3.47) 



We can also choose 



N-l / k 



2i-l 



^2W^) = g(^n2. + 2(M-Ar)_ 



^(2(M-A^))^2MX), 



(3.48) 



^We assume lI?'-^^^'^^\x) if n < 0. 
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but (p2N-i{x) is not a polynomial and needs to be treated separately. 
By the Rodrigues' representation 

and repeated integration by parts, we get for n > 



elf.2M.^^(2(M-iV))(2Mx)\ = 



1 + a V(^-^)+^ ^^^^, {n + 2{M-N) + l)\ _ (n + 2(M - TV))! 



2M ; V ^' (^-1)' 

(3.50) 

and 



eiT^^--, Lr"-^'^^(2Ma;) )^ = " ( ^ ) «(2(M - TV) + 1)!, (3.51) 



so that 



2M J {2j)\ 



n 



2k - 1 



L2A; + 2(M- AT) 



(3.52) 



and 



' ^2Mx ( \ 

' ' 4 



/ ,,^, (2i + 2(M-iV) + 2)! _ ,, (2i + 2(M-iV) + l)! - 
2M; (2j + l)! ^ (2j)! 

(3.53) 
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Now by the skew orthogonality, we can choose 

Af-2 ^ 

^2N-i{x) = -J2~ ((e^'^^",¥'2i+i(a;))^^2i(a;) - {^eir^'^^", (^2j(a;))^ <^2j+i(x)^ 

j=0 



and 



and 



and simphfy them separately. 



(3.54) 



"^^-^ ~[2M J " {2N-2)\ n 2k + 2(M - iV) " ^^'^^^ 
Now, we write 5*4 (x, y) as 5*40 (x, y) + 5'4f,(x, y), where 

iV-2 ^ 

54a(a;,i/) = ^ — (-z^2i(a;)^2j+i(l/) + ^2j+i(a;)^2i(l/)) (3.56) 



Sib{x,y) =^^{-i)'2N-2{x)i^2N~l{y) + ij'2N~l{^)i^2N~2{y)), (3.57) 



The formula (3.56) of our S4a{x,y) is also the formula for Si{x,y) in the LSE 
problem, with parameters M and N — 2, and has been well studied. For completeness 
we derive its Laguerre polynomial expression here, following [2]. 
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By the differential identity (3.43) and the identity 



nL(f(^-^))(x) = 

{-X + 2n + 2(M -N)- l)L^^[f-^^\x) - {n + 2(M - A^) - l)L'^^[f-^^\x), (3.58) 



we get, remembering the definition (3.19), the telescoping sequence 



^2,(^)=E n 



2i - 1 



k=0 \i= 



\2i + 2(M - A^) 



M-N + l/2-Mx + x^] l£("''-^))(2Mx) ) ^''-^-y^e-^'^ 



2i - 1 



j k 

2i + 2{M - N) 



i2k + l)Lgf--H2M 



x] 



k=0 i= 



-{2k + 2(M - iV))41*^^))(2Mx)) :,M-N-i/2^-M. 



2k - 1 



2 2fc + 2(M - iV) 



(2j + l)4?r''^^(2Ma;) 



a; 



M-Af-l/2 -Mx 



(3.59) 



and 



(^M - N + 1/2- Mx + x-^^ L^^l^l'^^\2Mx)x 



M-N-l/2^-Mx 



l[i2j + 2)L~mx) 



(3.60) 



(2j + 2(M - iV) + l)4f ^-^))(2Mx)) a;M-7v-i/2g-M._ 
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Therefore, if we plug in (3.47), (3.59) and (3.60) into (3.56), we get after some trick 



^4a(x,l/)=^(2M)2(^-^)+la;^"-^-l/2g-Mx^M-7V+l/2g-My 



'2N-2 

E 

j=0 



- {2{M-N)) 



(2Mx)Lf^'-^»(2My) 



{j + 2{M-N))\ ^ 



(2iV-2)! f'^'2j + 2{M-N)\ ^(2(Af-iv)) 



(2M-2)! 



n 



2j-l 



L 



2N-2 {2Mx)ip2N-2{y) } ■ 

(3.61) 



Furthermore, we can simphfy 'ip2N-2{x)- Since for j ^ 2N — 1, (if we define (pj{x) 



and then ^j{x) for j > 2N — 1 by the formula (3.45) and (3.46),) 



{lp2N-2{x)i^'jix) - i)2N_2{x)i)j{x)) dx = 0, 



(3.62) 



we get for j ^ 2N — 1, using integration by parts. 



^^^_2(x)Lf (*'-^))(2Mx)x^-^+^/2e-^^t/x = 0. 



(3.63) 



So by the orthogonal property of Laguerre polynomials, we get 



^^^_,(x) = CL?i'!^''^\2Mx)x'^^^~y^e-^^, 



(3.64) 



and we can determine that 



2 Al 2j + 2{M - N) 



(3.65) 
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without much difficulty. Together with the fact limj;_»oo '02iv-2(a;) = 0, we get 



N-l 



nh ('r\- 2iV-l -[-r 2 j - 1 Z""" ■M-jV-l/2^-Mt r (2(M- jV)) , . x , 

^2iv-2(a;) — 2j + 2{M-N) J, * ^ (2Mt)di. 

(3.66) 

Now, we can write S^aixjU) as S4ai{x,y) + S4a2ix,y), where 



j=0 



and 



/•CO 

4?J-2"'^^H2Ma;)a;^-^-V2e-^- / t^-^-^/'e-^*Lf^^-'^)H2Mi)di. (3.68) 
Finally, 

^4b[x,y)- gl^i + aj " (2M-1)! 

|45(fr''^^(2Mx)x^-^-V2e-^-V^2iv-i(z/) 

+ ^2iv-i(^) jj^" 45(i'r^^H2Mt)t^^-^-V2e-^*dt| , (3.69) 



and we can take the asymptotic analyses of S4ai{x,y), S4a2{x,y) and S4b{x,y) sepa- 
rately. 
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The same as in the complex case, we consider the rescaled distribution problem, and 
wish to find the probability of the largest sample eigenvalue being in the domain 
(0,p + qT]. We can put the kernel in the new coordinate system (after a conjugation 
by (^''o' g-1/2)), and get 

(P,.„,., <_,..r,'^- (. - (|«^;;^ 15;") ^ 

= det(/-PT(e,r7)), 



where as functions 



and 



SD^i^,?]) =q^SDi{x,y)\^=p+q(^, (3.71) 

y=p+qr) 

Si{^,r]) =qS4{x,y)\cc=p+qi, (3.72) 
y=P+qv 

ISi{^,r]) =IS4{x,y)\cc=p+qi, (3.73) 
y=P+qv 



In the proof of theorem L2, we need the matrix version of propositions |2.3| — 



2.6| and the fact that the convergence in trace norm of a matrix integral operator is 
equivalent to the convergence in trace norm of all its entries. 

Since the ISi{x,y) and DSi{x,y) are of the same form as S4{x,y), we only show 
the asymptotic analysis of S4{x,y), and state the result for the other two, for which 
the arguments are the same. We give proofs of all the three parts below. 
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In case — 1 < a < 7 we choose p = (1 + 7 and q 



7(2M)2/3 



and denote 







4/3 



(I + 7) 
7(2M)2/3 



,4/3 



x=(l+7-i)2 + -(l±^5 



(3.75) 



j/=(l+7-l)2 + 



7(2M)2/3 



V 



SAa{x,y) is the formula for the upper-left entry of the 2x2 matrix kernel of 
the quaternionic white Wishart ensemble with parameters M and — 1, and its 
asymptotic behavior is well studied [TT]. We want to prove that as M ^ 00, 5'4a(a;, y) 
dominates S4{x,y) in the domain that we are interested in, and so naturally the 
distribution of the largest sample eigenvalue in the perturbed problem is the same 
as that in the quaternionic white Wishart ensemble. (The difference between and 
iV — 1 is negligible.) 

SAai{x,y) is almost the kernel for the complex white Wishart ensemble with pa- 
rameters 2M — 2 and 2N — 2, besides a factor ^Jyjxj^. By arguments in subsection 
12.3.11 we have 

hm x(0^4ai(e,^)x(r/) = Jx(Oi^Airy(e,r/)x(r/). (3.76) 

M— +00 Z 



For the S4a2{x,y) part, we also have in trace norm [TT], 



lim xiOS,a2{^,v)xiv) = ~-;XiOm) / Ai(t)dtx(r/). 



(3.77) 



We get the proof of (3.77) by asymptotics analysis. Formula (5.165) and a similar re- 



sult for L^2n'-2 imply the convergence in norm of functions in ^ and respectively 



^Here * stands for 4, 4a, 4al, 4a2 and 46. The definition of S^,{£,,ri) in (3.751 is only used in 



subsection 3.4.1 and 3.4.2 
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lim 7-2^(1 + 7)^/3(2M)V3gM-iv^{2(M-7V))^2Mx)a; 



M-N-1/2 -Mx 



M^oo 



-2Nni\J^M-N 



lim 7-^^'2Me^' 

M^oo 



X(0 = Ai(Ox(0, 
(3.78) 

Ai(t)rftx(^), 
(3.79) 



and by the Stirling's formula, 



lim (2M) 



,^^_, (2iV-l)! 
(2M-2)!' 



2(Af-Af) 



(3.80) 



By (3.68) and (3.75), we get 



^-2^(1 + 7)4/3(2M)i/3e*^-^Lfi":V'^^^(2Ma;)x^'^-^-i/2e-*^^x(a 

7-2^2Me^-^ / Lfi";V'^^H2Mt)t^^-^-i/2e-^^*rftx(^7)- (3.81) 



Therefore we get the trace norm convergence (3.77) from the L convergence (3.78) 



and (3.79) by proposition 2.4 



Now we need to analyze the term 5'4b(^,?7), new to the perturbed problem. We 
need the following results of convergence, which are direct consequences of (5.165), 



(5.172) and a similar result of 'ip'2N-v 



lim 7-2^-1(1 + 7)^/3(2M)V3eA/-Jv^(2^M-A.))^2Mx)x^^"^-i/2e-*^^x(e) 



M^oo 



Ai(Ox(0, (3.82) 
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POO POO 

lim 7-2^2Me^^-^ / L^^^'';''^\2Mt)t''-''-'/'e-'^'dtx{v) = - / AKt)dtx{v), 

Jy Jr, 

(3.83) 



(3.84) 



(3.85) 



By the Stirling's formula, we get 



lim (2M) 

M^oo 



2{M-N) 



{2N 

(2M- 1)! 



,2(7V-M)y7V-l ^ -^^ 



(3.86) 



and then by (3.69), (3.75) and proposition 2.4, we have the convergence in trace norm 



r (1-«7)(2M)V=^ ~ w ^ 
(1+7)2/3 X(0g4.(e, ^)X(^) 



\x{i) (Ai(0 Ai(r^) + Ai'(0 M{t)dt^ xiv), (3.87) 



which implies that in trace norm. 



M— »oo 



(3. 



Now we get the desired result 



lim x{0S4^,v)xiv) = lim xiOSAai^,v)xiv) = xiOS4i^,v)x{v), (3-89) 

M— >oo M^oo 
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lim xiOSD.i^, r^)x{v) =x{OSD,{^, r/)x(r/), (3.90) 
lim x(0^4(e, V)X{V) =X{0IS,{^, V)X{V)- (3-91) 

M-+00 



Therefore, in trace norm 



and the convergence of Fredholm determinant follows. 
3.4.2 The a = 7^ part 



When a = 7"^, the 1 — 07"^ in (3.87) vanishes, so we need other asymptotic formulas 
for 'ip2N-i{i]) ci-iid 'ip2N-i{v)- The approach is similar to that in the a < 7"^ case, with 
the same choice of p and q. We need the convergence results given by (5.172) and 
similar results: 

lim 7-2^-^(1 + 7)4/3(2M)V3gM-Arg€/3^(2(M-N))^2Mx)a;^-^-^/2e-*^^x(0 = 

-e«/3Ai(0x(0, (3.93) 



hm 7-2^2Me^-^e-''/=' / Lfi''[''^\2Mt)t''-^-'/^e-'''dtxiv) = 

Jy 

POO 

-e-^l'' \ Ai{t)dtx{v), (3.94) 

J n 
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M^oo 

(3.95) 
(3.96) 



Now we conclude the proof of the = 7^ part of theorem 1.2 Using (3.69) 



(3.86) and proposition 2.4 we have the convergence in trace norm 



hm xiOe^^'S,,i^,r])e~'^/'xiv) =lxiOe^^' (^Ai(e)s(i)(r/) + Ai(0 ki{t)dt^ e~^'\{r^) 



(3.97) 



and this together with the conjugated convergence result of S'4a(^,?7) in formulas 



(3.76) and (3.77) of subsection 3.4.1 , which can be proved by arguments in subsection 



2.3.2, conclude 



hm x(e)e«/'^4(e, V)e-'^'\{V) = xi^e^^'S.i^, r/)e-^/\(r/). (3.98) 



In the same way we get 



hm x{Oe^^'SD,{^,r])e'^/'x{v) =x{Oe^^'SD,{^,r])e'^/'x{v), 



lim x{Oe~^^'lS,i^,v)e-''/'xiv) =x{Oe-^''lS,{i,r^)e"^/\{r^). 

M— +00 



(3.99) 
(3.100) 
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Then we get the convergence in trace norm of a conjugate of x{OPt{^,i])x{v) 



e«/3 \ (S,{i,n) SD,{i,r^)\ e-^/' 



xiv)- 



(3.101^ 



Therefore, 



hm det{I - PT{^,r])) 



(3.102) 



we we use the matrix version of proposition 2.6 



3.4.3 The a > 7"^ part 



If a > 7 ^, the location as well as the fluctuation scale of the largest sample eigen- 
value is changed. We change variables as p = (a + 1) ^1 + j and q = (a + 



1)^/1 — 7^^/^^ and then by (3.72) the kernel S^{x,y) after substitution is 



x=(a+l)ll+^) + {a+l) 1- 



?^=(a+l)(l+^)+(a+l)^l-^^r, 

(3.103) 



^Here * stands for 4, 4a or 4b, and the S^,{(,,ri) in this subsection is not identical to that in 



subsection 3.4.1 and 3.4.2 
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We analyze S^biC,,"!]) first. As before, we liave convergence results by (5.185) and 
similar results 

M^oo (72a)*^+^+l/2(a + l)A/-7V-l/2 V \ ' 

(3.104) 



1 (72a+l)^-^-i/2(7V-l) 



M "oo 2 (72a)^+^+V2(a + i)M-jv+i/2 V r " H^^^^ ; 



/■oo 



e-4 h^^Tip x(r/), (3.105) 



2 N M-N+l/2 2 2_;^ 



M-^oo V(7^a + l)(a + 1) 



"'/'V^27v-i(y)x(r/) = e 3 " "'/'x(^), (3.106) 



e 



o \ M-Af-1/2 9 2 2 

lim — r e . „ „ ^ — e 



M^oo V (7^0 + 1) (a + 1) y (72a2 - 1)M 

e-'«/'V^2iv-i(^)x(e) = e 5 (-^'"+1)^ ^ ^' xiO- (3.107) 



For notational simplicity, we denote functions on the left-hand sides of (3.104) 



(3.107) by Fi(0, i^2(r?), i^3(^?) and ^4(0, and denote 



CM = (2M)2(^^-^)|^-i|e^(^-^^)7^^-^ (3.10^ 
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4 



2 2 1 
7 a —1 



A/ 



.e(^^^T^*^(^-^)+^(«-^)/^F3(0F2(r^) , (3.118) 



with 

_(7V-1)3^V2M 

- a7(7^a + 1) " ^^'^^^^ 
Now we write Pxi^yr]) as the sum 

/V(e,r/) =P^,(e,r7) + /V,(e,r/), (3.120) 



with 



p tc \ f\l '^4a(^,^?) SDiai^,ri) /onn 



If we denote 



[7(0 = 1 2 , I, (3.123) 



[/-i(r^) = 1 2 F3(0^^ I (3.124) 
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Cm 
2 



uiOPTti^.v)U'\v)2i -^F.io {fM + ^^^^ 



with the entry 

^^'^ ^e~i^^^^^'''^''/'F,iOFs{v)-Fs{OF2{^^^^ 



C, 



M 



xiv), 

(3.125) 



(3.126) 



We want U{^)PTb{C,,i])U ^{r]) to converge in trace norm as M — »• cxo, and need 
the result 



Lemma 3.2. In L? norm, 



g 4 (7^a + l)^ « X(6- 



(3.127) 



The proof is left to the reader. The main ingredient is (3.105) and the fact that 
Fa{.OIF^{^) approaches to 1 uniformly on [T, oo). 

We need another convergence result on U {^)PTai^, (rj): 

Proposition 3.1. In trace norm, 



hm U{OPTai^,v)U~\v) = 0. 

M— >oo 



(3.128) 



The proof is left to the reader. Since all the four entries in PTa{C,,f]) can be 



expressed by Laguerre polynomials, the asymptotic results like (3.104) and (3.105) 



give the convergence (3.128) 
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By lemma 3.2 and proposition 3.1 we get in trace norm 



lim det(/- Pt(^,^)) 

M— >CXD 

lim det{I-U{OPT{^,v)U-\v)) 
lim det{I-UiOPTb{^,v)U~\v)) 

M— +00 



e 2 dt 



(3.129) 



and we get the proof of the a > 7 ^ part of theorem 1.2 



Chapter 4 



Phase transition phenomenon 



4.1 Rank 1 complex spiked model 



Here we assume that the single spiked population eigenvalue isl + a = l + 7 ^, and 



by the part 2 of theorem 1.1, we know the distribution function of the largest sample 



eigenvalue is -Fguei, which is, according to (1.67), defined as 



GUEl 



(T) = det (1 - xiO {KAiry{^,v)+m>^'\v)) Xiv)) 



(4.1) 



Forrester recognized that 



GUEl 



(T) 



(4.2) 



Therefore as the perturbative parameter a increases, by (1.70) and (4.2) we have the 



-^GUE — -^GOE — Gaussian phase transition phenomenon around a = 7 ^. 
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4.2 Rank 1 quaternionic spiked model 



Again we assume that the single spiked population eigenvalue isl + a = l+ 7 ^. As 
the perturbative parameter a increases, we have the -Fgse — -^goe — Gaussian phase 



transition phenomenon around a = 7 ^, by results of theorem 1.2 and 1.3 In this 



section we prove theorem 1.3 



In manipulation of kernels, we follow the method of [29]. The procedure seems 
informal and cursory, but is carefully justified in [29] . 
For notational simplicity, we denote = X{t,oo)(0) 



POO 



(4.3) 



First, we express the integral operator 



y{0'iS4{^,v)x{^) xiOMv.Uxiv) 



(4.4) 



by 



xiOi \ flS,{^,v)x{v) S,{r^,Ox{r]) 
xiO [TS,i^,v)xiv) S,ir^,^,)xiv), 



since by (3.34) - (3.37) and taking limit 



d_ 



d_ 



(4.5) 



(4.6) 
(4.7) 



Then using (3.29) for A bounded and B trace class, upon suitably defining the Hilbert 
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spaces our operators A and B are acting on, we find 



det \ I- MV,0X{V)] (xiv)l 

■det I / - ^4(^?,6x(^) 
,/^4(C,r?)x(r?)|^ :54(r?,C,)x(r?)^ 



and by conjugation with j ^ ^ ) , we get 



^ \ 

= det ( / - v)xiv) ^ + ^4(r/, Oxiv) 



)) 



(4.8) 



Since 



ri=oo 



d 



n=T Jrp drj 



IS,{C,v)mdv, (4.9) 



as an operator 



IS4{^,V)X{V)^ = IS,{^,oo)SM-IS,{^,T)St{v) - ^IS,{^,v)x{v), (4.10) 



where S^o and St are (generalized) Dirac functions. Then with the help of identity 



roo poo rcsD poo 

/ KAiryit, V)dt+ KAiry{C,t)dt = Ai{t)dt Ai{t)dt, (4.11) 

Jf) 
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which can be proved directly from (1.61), we get 



+ 



Q 1^ KA.y(e, t)dt - \b{t)b{0 - Ib{0 + Ib{t)^ 6t{v) 



(4.12) 



Now we denote R{^,ri) as the resolvent of Kj<^ijy{^,ri)x{'r]), such that as integral 
operators 

I + R{^,V) = {I-KMry{^,r])x{v))-\ (4.13) 

then 

=(/ - irAiry(e, v)xiv)) + R)ii - ^BiO) Hv)xiv) 

+ {I + R) Q 1^ irAiry(e, t)dt - \b{T)B{0 - Ib{0 + Ib{T)^ 6t{v) 
+ ^-{1 + R)Bi05ooiv) 



Again by the formula (3.29), in the form of (formula (17) in |29j ) 



det J - ^ ttfc ® = det (5,, - (a„ A)),- ^=1,...,, (4.15) 



fe=i 
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we get 



det{I-{I + R){l--B{0)Ai{ri)x{v) 

+ {I + R)(- i^Airy(e, t)dt - -B{T)B{0 - ^B{0 + ^B{T) ) Sriv) 



+ -{I + R)B{OSM 
/l + ail Oii2 



= det 



Oils 

a21 1 + 0(22 Oi23 
\ (^31 Ol32 l + a33; 



(4.16) 



where upon the definition 



/oo 



(4.17) 
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we have 



an ^(^{I + R){1- Ib{0), - Ai(0)^ , (4.18) 



Q:i2 = 



Q!l3 



hi + R)B{0,-Mo) , 



I T 



a,, = {I + R){1 - -B{0) 



a,, = {I + R)[- KA^ii, t)dt - -B{T)B{0 - -B{0 + -B{T) 



a2s = ^{I + R)B{0 



as,^(I + R)(l--B{0) 



(4.19) 
(4.20) 

(4.21) 
(4.22) 
(4.23) 

(4.24) 



«32 = (/ + i?) ( ;^ / i^Airy(e, t)dt - -B{T)B{0 - "5(0 + -B{T) 



ass^-{I + R)B{0 



0, 



(4.25) 
(4.26) 



If we take elementary row operations, we get 



det 



:det 



:det 



^1 + ail Oii2 ai3 ^ 

a2l 1 + a22 0(23 
\ Oisi a32 1 + Q;33/ 

^l + aii-ai3 ai2-\B{T)ai3 

0L2\ — OL23 1 + OL22 " \B{T)a23 OL23 

^ 1 



11 



12 



/5: 

1 + A 



(4.27) 



'22 , 
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where 



/3n=((/ + i?)(l-5(0),-Ai(e)) 



(4.28) 



(3,^ = {I + R){1-B{0\=T' 
1 

2' 



/?22 = ^{I + R){ I K^iry{U)dt - B{T)B{0 - B{0 + B{T) 



T 

(4.29) 
(4.30) 

(4.31) 



Using (4.13) and (4.15), we observe {s^^KO = 1 - 5(0) 



det(J - i^Airy(^, v)xiv)) det 



l + Pu P12 

[321 l + /?22, 



det ( / - {KAiry{^,r^)x{v) + s^'\0 Hv))x{v) 



+ 



\ (^j^ KMryi^, t)dt - B{T)B{i) - B{i) + B{T)^ 6t{v) 



(4.32) 



If we denote R{^,r]) as the resolvent of {KAhyi^,v)xiv) + ^^^KO ■^Kv))xiv)j so 
that as operators 

I + mv) = {l+{KAiry{^,v)x{v) + s^'\OHv))xiv)y\ (4.33) 



and 



QiO = {I + R){ 1^ KAiry{^,t)dt - B{T)B{0 - B{0 + B{T) ) , (4.34) 



then 



i^GSEi = det (J - {KAir^{^,r^)x{v) + s^'\0 Hv))xiv)) det (l + ^Q(^)St{v)) ■ 



(4.35) 
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To prove theorem 1.3, we need only (4.1) and (4.2) with ^ and 1] swapped, and 



det [ I + -QiOSriv) ] = 1, 



(4.36) 



which by (4.15) is equivalent to 



Q{T) = 0. 



(4.37) 



If we take /(O = Q(0 + 1, then (4.37) is 



(/ - {KA.y{^,v)x{v) + s«(0 Ai(r/))x(r])) {f{0 - 1) = 

/oo 
KAiryi^,t)dt-BiT)B{0-B{0 + BiT), (4.38) 



which is equivalent to 



(4.39) 



The integral equation (4.39) is solvable, and the solution is 



l-{{I + R)sW{0,Ai{0)T 



(4.40) 



Therefore to prove the theorem (1.3) we need only to prove f{T) = 1, which is 
equivalent to 



(/ + R)s^'\T) = 1 - ((/ + i?).(i)(0, Ai(0)T. 
This is a nontrivial result, but it can be derived by results in [2S], with[^ 



(4.41) 



q(s)ds 



{I + R)s^^>{T) =e-JT 
((/ + i?)s(^)(0,Ai(0)T =1 - e- 



(4.42) 
(4.43) 



^In section VII of [29] Tracy and Widom define function q and u for both GOE and GSE. Our 
(/ + i?)s(i)(T) is equal to times their q in GOE and our ((/ + i?)s(i)(6> Ai(0)T is equal to 2 
times their u in GOE. 
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where q is the Painleve II function described in (1.32) and (1.33) 



We can give a proof of (4.42) and (4.43), based on the method and results in 



First, assume T is fixed, then (J + is a function, and we have 



d 



[l + R) 



s^'\0- (4.44) 



Since = Ai(0 and we have (2.13) in [2H], which is 



d 



[l + R) 



(2 + R) Ai(e) ■ (1 - Ky\Ai{r])x{v)) + R{v, T) ■ p{T, r^), (4.45) 



where p{x, y) = 6{x — y) + R{x, y) is the distribution kernel of 1 + -R, and is the 
transpose (as an operator) of i^Airy(^, ''7)x('7), we have 



d 



{i+R)s^'\o = {^+R)m) 

-{l + R) Am ■{(! + R)s^'\0, Ai(0)T + Ri^, T) • (1 + R)s^'\T). (4.46) 



If we regard T parameter, then we have 



dT 



(/ + i?)s«(e; T) = -i?(e,T) ■ (1 + R)s^'\T), 



(4.47) 



because (2.16) in [28] gives 



dT 



:i + R) = Ri^,T)-piT,r^). 



(4.48) 



Therefore, if we set ^ = T and take the derivative with respect to the parameter T, 



CHAPTER 4. PHASE TRANSITION PHENOMENON 



91 



we have 



d 



(4.49) 



;i + i?)Ai(r)-(i-((j + i?).W(0,Ai(0)T) 



On the other hand, by (4.47) we have 



dT 



((/ + i?).W(e),Ai(0)T 



d 



[1 + R)sW(T) . Ai(T) + ( — (/ + i?).«(0, Ai(0)T 
:i + R)s^'\T) ■ (^Ai(T) + 1^ i?(e, T) Ai(Orfe) 
[l + R)s^'\T)-il + R)Ai{T). 



(4.50) 



(1.11) and (1.12) in [28] give the result 



:i + R)Ai{T) = q{T), 



(4.51) 



and now we if we denote (J + R)s^^\T) = st and ((/ + R)s^^\^), Ai{^))T = wt, we 
have 



d 
dT 



St =q'(l — Wt) 



d 

dT 



(4.52) 



1 - Wt) =qsT- 



Now we can get (4.42) and (4.43) by boundary conditions. 



4.3 Conjectures of phase transition in quaternionic 
and real spiked models 

In the complex spiked model, we have more complicated phase transition phenomenon 
for the limiting distribution of the largest sample eigenvalue, if the rank is greater 
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than 1. For example, if there are two spiked population eigenvalues l + cti and l + a2, 
we have the phase diagram as ai and a2 vary from —1 to oo: 



7 



-1 



-^GUE 



Gaussian 



^GUEl 




7" 



GUE2 



Gaussian oii 



-^GUEl 



Figure 4.1: Phase diagram of rank 2 complex spiked model 

Analogously, we conjecture the phase diagram for the rank 2 quaternionic spiked 
model for the limiting distribution of the largest sample eigenvalue, with spiked popu- 
lation eigenvalues l + cti and l + a2- The Gsi is the distribution function of the largest 
eigenvalue of a 2 x 2 random quaternionic Hermitian matrix ( c+-id+je+kf ^ ^ 
where a, . . . , f are independent normal random variables with mean 0, the variance 
of a and 6 is 1, and the variance of c, . . . , / is 1/2. Our conjecture for Gs2 is based on 
the pattern of Gt for the rank t complex spiked model: Actually Gt is the distribu- 
tion function of the largest eigenvalue of at xt random Hermitian matrix (ajj)i<ij<t, 
where an, ^{uij), ^{aij) {i < j) are independent normal random variables with mean 
0, the variance of diagonal entries is 1, and the variance of real and imaginary parts 
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Figure 4.2: Phase diagram of rank 2 quaternionic spiked model (conjecture) 



of off-diagonal entries is 1/2. Therefore we guess such distribution function for rank 2 
quaternionic spiked model should be the distribution function of of the largest eigen- 
value of a t X t random Hermitian matrix. It is obvious that in the rank 1 case, 
the Gaussian distribution satisfies the conjecture trivially, and the next distribution 
function is Gs2- But what is -Fgse2? Is -Pgse2 ^ Fredholm determinant? Our only 
clue is that the -Fgse2 should be similar to the Fq*^™ defined in p^ . 

For the real spiked model, even the rank 1 case is speculative. We conjecture that 
if the only spiked population eigenvalue is 1 + a, then the limiting distribution of 
the largest sample eigenvalue has the pattern Fgoe — -^gse — Gaussian. The Gaussian 
part has been proved in [23j, and other results are missing. 



Chapter 5 



Asymptotic analysis 



In sections 5.1 5.3[ the convention of notations is the same as that in chapter |2| e.g., 



ipr' is defined by (2.77). In section 5.4, the convention of notations is the same as 



that in chapter 3 e.g., ipr' is defined by (3.19). 



5.1 Asymptotics of tlj{p -\- q^), t/Jr'ip + qO^ ip{p + Qv) 
and tlJr'{p + qr]) when a^/ < 7"^ 

In this section, we assume x = p + and y = p + qr], where p = (1 + 7~^)^ and 

q - ^jvf2/3 • 

For the asymptotic analysis, we define S°° = Sf^ U U S^, where 



={te^ \ t>l}, 



4 2 

1 I 2 _ _ gj, 

--{-te^ I t < -1}, 



^2 



y-ioo 
^3 



(5.1) 
(5.2) 
(5.3) 
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and (c < 0) 



--{w G I ^{w) > c}, 



(5.4) 
(5.5) 



We have 




27rz 



Figure 5.1: 

e^'^-'^du = Ai(0, 



since by the substitution u = iv, we get (jgogSTri/e is defined below 1.62) 



27r 



g57ri/6 



(5.6) 



(5.7) 



which agrees with the integral definition of the Airy function, with the integration on 
the right hand side from ooe^'^*/® to ooe'^^^^. By direct calculation, we also have the 
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result that for any T, if — c is large enough 



27ri 



3 du 



< 



— c 



(5i 



Similarly, we define r°° = Tf^ U U T;^, where 



={-teT I t < -1} 
{te^ \ t>l} 



poo 
2 



poo 
^ 3 



- < t < -}, 

3 - - 



(5.9) 
(5.10) 
(5.11) 



and 



<c 



poo 
^ >c 



={w G r°° I ^{w) < c}, 



oo 



(5.12) 
(5.13) 



We have (similar to (5.6)) 



(5.14) 



and for any T, if c is large enough 



27ri 



3 du 



< 



(5.15) 



5.1.1 Asymptotics of i/^ip + q^) and i/^r'ip + 

We only analyze "^Ar' (p + 9^) • The analysis of ipip + is similar and simpler, and we 
only give the result. 
First we have 

,M.. (izil!^ = e-^'^f^^^-^^^^'^'^^, (5.16) 



2; 
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Figure 5.2: T'^ 



where 



f{z) = -(1 + i-'Yz + \ogz- iog(^ - 1), 



(5.17) 



and here and later, we do not need to concern ourselves about the ambiguity of the 



value of logarithmic functions. Now we can write (2.77) as 



1 



n 



1 + 



1 + as 



t'~i 



dz. 



(5.1^ 



For f{z) we have 

• J [z) = , with the zero pomt z — — 



z z-1] 



7+1' 



CHAPTER 5. ASYMPTOTIC ANALYSIS 



98 



7 + 1 
,// / 7 



7 + 1 



0; 

. 2(7 + 1)^ 

^3 



> 0. 



Hence locally around z = 



/ — 

where 



+ w 



7 + 1 

7 



(7 + 1)^ 



+ log 7 - (1 - 7^) log(7 + 1) + 7 TTi + ^ / + Ri {w 



373 



= 0{w^), as w ^ 0. 



(5.19) 



(5.20) 



After the substitution z = w + we get by (5.18) 



1 f Af(^^-log7+(l-7-')log(7+l)-7-'«-^^«'^-Ri(«')) + ^i±^Ml/'€(«'+:^) 



)r—r 



W 



1 



a,- - 7 



n - + 7TT^ 



7+1 / \J=1 



(l+7-i)(l + a,-) 



w + 



as' - 7 



-1 



(l + 7-i)(l + a,0 



t'-i 



dw 



(-1)^ (7+ l)^-^^^Mx i Ji+7//^MV3g^-i^M^3-Mfi,M 



27ri 7^-^ 
w + 



)r— r' 



7+1 



0=1 



nl ~ 1 

. , . . ^ (l + 7-^)(l + a,) 



-1 



w + 



fls' - 7 



-1 



(l + 7-i)(l + a,0 



(5.21) 



where is a contour around w = composed of S]^, E^, and S^, which 
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are defined as 



M 



2^2 — 



0' 27ri 

t — —e.— 
7 + 1 

7 

(7 + 1)4/3 



(7 



^M-V3 < ^ < 4} , 



4 2 
- < t < — 
3 - - 3 



7 + 1 



7 



7 + 1 



< t < 4 



2\/3 



(7 + 1)1/3 



(7 + 1) 



< t < 2\/3 



(7 + 1) 



For asymptotic analysis, we define 




Figure 5.3: E 



^remote — 1^1 ^3 M ^local" 
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Now if we denote 



7 



we have by (5.21 ) 



7 



(7 + 1) 



A/-Af 



e T+i 



and establish several lemmas: 



Lemma 5.1. IfT is fixed and M is large enough, 



2m 



^4 



FM{i,w)dw 



< 



1 e-g/^ 
3 MV40 ^ 



/or any ^ > T. 



Proof. By (5.17) and (5.19) 



(7 + ir 



+ Ri(w) 



-f 



7 , \ , 7 



7 + 1 
^7 + 1 
V 7 



7 + 1 



log7 + (l-7 ^)log(7 + l)-7 "^Tci 



w + log 



7 + 1. 
7 



w + 1 ) — 7 ^ log ((7 + l)w — 1) — 7 "^iri. 
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If w e Ef , 3fJ(w) = -2;^, and denote 9 = arg(w) e [f , f ], we have 



=2-^ + log (V(-l)2 + (2tan^)2) - 7-2 log (^^(27 + 3)2 + (27tan^)2) (5.32) 
>2^^ + 0-7-2 log (^^1672 + 127 + 9) , 



7 + 

and we can prove that if 7 > 1, 



2-^ + 0-7-2 log (^^167^ + 127 + 9) > 2 - log V37 > 0. (5.33) 

Therefore on Ef , if ^ > T, for < e' < 2 - log V37 and M large enough, 

\FM{tw)\ 

(^+1)4/3 
7 

)r—r' 



< 



r 



0=1 



(i + 7-i)(i + %); y V (i + 7-^)(i + «s') 



^g-2(e-T)(7+l)i/3Mi/3^(logV37-2+e')M 



If M is large enough, 



(5.34) 



g(IogV37-2+.')M <^!L_le-r/2^-l/40^ (5 35) 

^ 7+1 

g-2(^-r)(7+l)^/3MV3 <gT/2g-e/2^ (5_3g) 

and we get the result, since 

FM{C,w)dw 



1 



< — ^ max|FM(C,HI (5-37) 



27r «;esf' 
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□ 



Lemma 5.2. IfT is fixed and M is large enough, 



2m 



yM 
remote 



FM{i,w)dw 



< 



(5.38) 



for any ^ > T. 



Proof. For w G S^/^^^^g, we denote / 



-3?(w) = "Y^. Since arg(w) = we get by 



(5.31) 



3fJ 



373 



u;^ + i?i(ti;; 



(7 + 1)^ 
1' 



2 \ 7 V 7 



^log (l + 2(7 + l)/ + 4(7 + l)¥). 

(5.39) 



Then we take the derivative on both sides of (5.39) 



di V 373 



w'^ + Ri{w) 



, (7 + 1)^ 

^3 



1 + (7-1)^/ + 27 (^-^i) 



1 _ 2i±i/ + 4 h^i) ) (1 + 2(7 + 1)/ + 4(7 + lyp) 



(5.40) 



and are able to find a positive number e", such that for < / < 2—^ 



, (7 + 1)^ 

^3 



1 + (7 -1)^^ + 27 {^l 



1 _ 22±1/ + 4 (^/) J (1 + 2(7 + 1)/ + 4(7 + im 



> 3e"l\ (5.41) 
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and on the two right most points of ^fLote^ {-l + VS)M-^°/^^ and {-l-VS)M-^°/^^, 



373 

8(7+ir 
"3 73 

8(7+ir 



3 73 

> / 3e"t^dt 
Jo 



«i=(-l±V3)M-io/39 
^-10/13 ^ O(M-40/39) 

^-10/13 ^ O(A/r-10/39)) 



(5.42) 



Hence we know that for w G S 



M 

remote ' 



3? ( ^^^^^-^Mw^ + MRi{w) 1 > M 



AflO/39 



3e"t2rft = Mt'l 



(5.43) 



and have the estimation that if ^ > T, for < e'" < e" and M large enough, 

(/ > M-lO/39) 



7 



' n • ^+ 



w 



7+1 y 

U/3 



= 1 



a. - 7 



(l + 7-i)(l + a,) 



w 



as' - 7 



(l+7-i)(l + a,0 



t'-i 



(5.44) 



Now we get the result by similar inequalities as ( 5.35 )-( 5.37). 



□ 



If we define 



Cr' 



n 



aj - 7- 



L V(l + 7-^)(l + a,); J V(l + 7-^)(l + a.') 



as' - 7 



-1 



(5.45) 
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we have 



Lemma 5.3. IfT is fixed and M is large enough, 



2m 



local 



FM{tw)dw - (-1)V"'''(7 + Ai(0 



1 e-«/2 
< SJiWo^ (5-46) 



for any ^ >T. 



Proof. On Ej^^^i, \w\ < 2M-^^/^^, and Ri{w) = 0{M-*^/^^), so that 



(7+1) 



(1 + C>(M-V39)) , (5 47) 



% - r 



as' - 1' 



t'-i 



^ 373 



7 



and the C(M ^/^^) term is independent of ^. After the substitution u — ' M^^^w, 
we get 



27ri 



local 



FM{i,w)dw 



2m 



e^'^-'^du (1 + C>(M-V39)) _ (5_4g) 



>- (l+7)^/% l/13 
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On S°°, if ^ > T, |e(«-^)"| < e^/^e~^/^, and we have 



1 

27n 



local 



T/2-(/2 



FM{^,w)dw - (-1) V-^'(7 + ly'Cr'-i Ai(0 

(-i)v-^''(7+i)^''a'-i 



27ri 



e » 



<_(7+]y^j(^l/13 

- 7 



(-l)V"'-'(7 + ir'a.'-i 



2ni 



Tu- — 

e 3 



duO{M- 



-l/39^ 



(5.49) 



and we can get the result by direct calculation. 



□ 



By lemmas 5.1-5.3, and (5.29), we get the convergence result 



7 ^ ' (7+1)" 



(-i)V-^'(7 + i)'^'a.'-iAi(o 



< 



jVfl/40 



. (5.50) 



In the same way, we have the result for ip{p + qC) 



(7 + ^Y'\il^_^^N _^ 2^!^e-^^"7/>(p + gO - (-7)'' Ai(0 



7 



(7 + 1)*^ 



M-N 



< 



MV40- 
(5.51) 



5.1.2 Asymptotics of i/j{p + qr]) and + qv) 

We only analyze ipr'{p + Q''?)? The analysis of iplp + gr/) is similar and simpler, and we 
only give the result. 
We have 



e -^--^ = e^/(^)-^^^^V^ (5.52) 



z-1 
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where f{z) is defined by (5.17). Now we can write (2.78) as 



1 

27Ti 



yr—r'+l 



n 



z — 



1 + Cts 



-t' 



dz. (5.53) 



After tlie substitution z = w + we get 



1 r Af-(^2±l+iog^-(l-^-2)log(7+l)+7-Vi+(^«;3+^^(^)^_(i±l)l^A^^ 



2m 



w 



7+1 



,-1 



1(7 + 



as' - 7 



-1 



7+1 



ttA^x ^ 

-P 7 + 1 (I) P I 

iTll (7 + 1)^-^ /fM 



l + 7-i)(l + a,0 



dw 



W 



7+1 



W + 



7+1 



r— r'+l 



n 

.i=i 



w 



a,- -7 



-1 



(l + 7-i)(l + a,) 



as' - 7^" 



(l+7-i)(l + a,0 



(5.54) 



wliere is a contour containing and (j = 1, • • • , s), composed 

of rf , rf , rf , rf , rf and rf , wliicli are defined as below, witli tlie constant 
Cright a large enough positive number, so that f *^ contains all the poles if M is large 
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enough. 



M 



^ 1 



pM 
^ 2 



pM 
^ 3 



7 / 7 + 1 



e 3 



< t < - - 



7 (7 + 1)'/' 



M-V3 



(7 + 1)^/' 



TT TT 
< t < - 

3 - - 3 



0' Svri 

i — 



7 + 1 



7 



(7+1)1/3 



-t + 



t - 



V3 , 
7 + 1^ 

V3 



~ Cright ^ t < 



7 + 1 ' 



7 + 1 

C'right -'r it 



7 + 1 
V3 



< t < Cright 



(7 + 1) 

For asymptotic analysis, we define 



< t < 



V3 



(7+1) 









pM / 14 
\ 2 


pM 




—7 i 

i 3 \ pM 

\ ^- 









Figure 5.4: F 



M 



(5.55) 
(5.56) 
(5.57) 

(5.58) 
(5.59) 
(5.60) 



fiscal =U e r^^ I ^{z) < M-'°/39}, 



pM (r^ I I p-'^^ \ 

remote — 1 ^ 3 M ^ local" 



(5.61) 
(5.62) 
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If we denote 



7 



( 



)r— r'+l 



n -+71 



% - 7 



(l + 7-^)(l + %) 



w + 



Os' - 7" 



(l + 7-i)(l + a,0 



(5.63) 



we have 



■1) 



^(7 + ir-^„^M,~ , , 1 



7 



M 



_e7+i ^ipr'ip + qri) 



2ni 



GM{V:t^)dw, (5.64) 



and establish several lemmas: 



Lemma 5.4. IfT is fixed and M is large enough, 



2m 



rf urf ur^ 



Gm{v, w)dw 



< 



1 e-W^ 



(5.65) 



for any rj >T. 



Lemma 5.5. IfT is fixed and M is large enough, 



2m 



pM 
remote 



< 



1 e-'i/^ 
3 MV40 ' 



(5.66) 



/or any rj >T. 



Lemma 5.6. IfT is fixed and M is large enough. 



1 f f-D'-Hi + 7"M 

— / FM{v,w)dw- \J Aiiv) 
2m JfM 7^ (7 + 1)^ Cr' 



< 



1 e-V^ 
3MV40' 



(5.67) 



for any rj >T. 
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Since the proofs of lemma 5.4 5.6 are similar to those of lemmas 5.1 5.3, we only 



give an outline of the proof of lemma 5.6 



Sketch of proof of lemma\5^ On T^^^^, \w\ < 2M-^^I^^ and Ri{w) = 0{M-^"/^^), 
so that 



after the substitution u = (^+"^) M^/^w, we get 



<(l±lll^ A/1/3 



e-^"+^rfM(l + 0(M-i/=^9)). 

(5.69) 



Also we have that on r°°, if 77 > T, |e "^^^ < e^'^^e ''''^|. We can prove lemma 



5.6 



in the same way as proving lemma 5.3 



□ 



By lemmas 5.4-5.6, and (5.64), we get the convergence result 



-ir-i(i+7-^: 



< 



MV40 



Ai(0 
(5.70) 



And in the same way, we have the result for ip{p + qC,) 



(7 + I) 

7 



4/3 



(7 + 1) 



M-N 



eTr.^y^{p + qr^) - [-iy~'^~- Ai(0 



7 



M 



< 



e-v/2 
MV40 • 
(5.71) 
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5.2 Asymptotics of ipr'{p~^QO i^r'iP ~^ QV) when 



a,/ = 7 

In this section, we still assume x = p + q^, y = p + qr], p = (l + 7~^)^ and q = '^l^jl/i ■ 
With defined in last section, we have 



2m 



3«"-^^z*'-Mn=(-l)*'-V*')(0 



(5.72) 



which can be proved by a simple change of variable similar to (5.7). For any T, if — c 
is large enough 



< 



— c 



we define r°° = U U T;^, where 



(5.73) 



rr ={-tef I t < --}, 



poo 
J- 3 



--{te- \ t>-}, 



(5.74) 
(5.75) 
(5.76) 



and 



f- ={w e I ^(w) < c}. 



>c 



(5.77) 
(5.78) 



We have 



1 

27ri 



-r)u+ 



du 



(5.79) 
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Figure 5.5: 



and for any T, if c is large enough 



27ri 



3 du 



< 



5.2.1 Asymptotics of il^r'{p + Q^) 



Similar to (5.18), we have {f{z) is defined by (5.17)) 



2m 

iz- 



iFini^-iT^ 
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and after the substitution z = w + we get 



2Tii JgAf 



(-1)^ (7 + 1)^'^-^ ^^Mx I J^±4!1m^/Hu,-^^Mu,^-MR,{u,) 



2'Ki 7*^ 
w + 



0) e ^ 



r— r 

T- ^ /s'-l 

7+1 



«i - 7 



(l + 7-i)(l + a,) 



(5.82) 



with S^^ defined by ( 1^22] ) -( [K25| ). 
Now we denote 



FMt'i^,w) 



7 



)r—r 
' n f ^+ 



w — 



7+1 



o=i 



«i - 7 



-1 



(l + 7-^)(l + a,) 



w 



.t'-i 



(5.83) 



and have 



(5.84) 



(7 + 1)*^-^ 
Similar to the a^' < 7^"*^ case, we have 



Lemma 5.7. IfT is fixed and M is large enough, 



2m 



< 



MV40 ■ 



(5.85) 
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Proof. Similar to the proofs of lemmas 5.1-5.3 together 



□ 



Hence we have the convergence result 



7 



e e' 



-«/6 



< 



MV40 • 



(5.86) 



5.2.2 Asymptotics of i^r'ip + Qv) 



Similar to (|5.53|), we have 



1 



gA/ /(^)- {Z - 1) 



^ n 



+ ai 



1 



7 + 1 



dz, (5.87) 



and after the substitution z = w + we get 



1 f A/-(^^+log7-(l-7-2)log(7+l)+7-2^i+^^«'H^?i(«'))-^i±^Ml/3^(^+_i_) 



27rz 



(--^^ 




r 


(- + ^Jt) 

(-1)" 


r-r'+l 


27ii (7 + 1 

- +) 


)A/- 
r 




7 — 


r'+l 



a, - 7- 



-^A^-a; 
_g 7+1 



(l + 7-i)(l + a,). 



pM 



= 1 



a. - 7 



(l + 7-i)(l + a,) 



(5. 
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where is slightly different from T^: is composed of Tf, . . . , F^, which are 



1 



--^M 



7 TTi 

-e 3 



7 V 7 + 1 
7/6 



< t < 



1/6 



7 (7+1)'/^ 



(7 + 1)^/^ 



1 5 
3 - - 3 ^ ' 



7 + 1 



1/6 ,,1/^ 2 



(7 + 1)1/3 



W -pM 



Ff, for * = 4, 5, 6. 



7 



(5.89) 

(5.90) 

(5.91) 
(5.92) 



We also define 



^ 2 








pM 
^ 6 




r 1 

I3 \ rf 

^ B"^ 


[ ► 







Figure 5.6: F 



M 



local 
remote 



= (ff U f3^) \ fi^,,. 



(5.93) 
(5.94) 



If we denote 

" fn + °'Tj:' w-'', (5.95) 
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we have 



1) 



and 



TV 



(7 + 1)*'-^ 



7 



M 



(7 + 1) 

7 



4/3 



-Ml/3 



27ri 



pM 



(5.96) 



Lemma 5.8. //T is fixed and M is large enough, 



^1 GM^\v.^v)dw-4^P^^^{-lY-h^Hv) 
2m JfM Y (7 + 1) Cr'-t' 



< 



Ml/40 ' 



(5.97) 



for any rj >T. 



Sketch of proof. The integral of GMt'ijIi''^) over F*^ \ T^^^^i is neghgible, and we can 

and get the same resuh. On Vf^^^^i V ^ T, 



estimate it as lemmas 



5.4 



and 



5.5 



|g iv T)u^ ^ g T/6g?7/6^ g^j^^ pg^]^ carry out the proof like that of lemma 5.6, with 
the upper bound of |e~ "^-'"| changed from e-^/^e"''/^ to e~'^/^e^/^. □ 



Therefore we have the convergence result 



7 



(^ + 1)'^' mV3^ (-l)^ ^^ + y"'' e^^%-'/3V^.,(P + ?^) 



7 



(-1)^(1 + 7-^) 

7^-^'(7 + i)'-'a'-t' 



Nt'-l --r,/3 ft') 



< 



MV40 



. (5.98) 



5.3 Asymptotics of tlj{p + g^), + Q'f), + qr]) 

and '0^/(2? + qr]) when a^/ < a 

In this section, a is a member greater than 7"^, and we assume p = (1 + a) ^1 + 
and g = (1 + a)-i/l 



1 1 
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— 1 — oo <t< oo}, 


(5.99) 




—{w 


e E°° 1 1^1 < c}, 


(5.100) 








(5.101) 



and for any T, if c is large enougli, 



Figure 5.7: E= 



Figure 5.8: 



1 



< 



We also define 



poo ^ |git/3 I < t < 27r}. 



(5.102) 



(5.103) 



We need two integral representations of Hermite polynomials. First, we have the 
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explicit formulas for Hn{x) 



H,A^) =i-l)H2k - 1)!! I 1 + (-2^(-2>. + 2).., (-2>. + 2,-2) ^,, 



1=1 



(5.104) 



(2j)! 



On the other hand, {v = u + C,) 



(5.105) 



— / 2 M =e 2 — e 2 u ^du 

"004+^ 2 



1 

=e 2 

2m 



=e 2 



1 

2^ 



e 2 (ti - ^)* 



(5.106) 
(5.107) 



where the integral in (5.106) is along the vertical line parallel to through the 



point f = ^, and the equivalency of (5.106) and (5.107) is a simple application of the 



Cauchy integral formula. Now if we write 



27ri 



27ri 



e'^v^dv, (5.108) 



and 



/ e 2 v-^dv = — 



e 2 x^dx 



(-1)* 



j odd, 

{2k - 1)!! J = 2k. 



(5.109) 



Compare (5.107), (5.108) and (5.109) to (5.104) and (5.105), we get 



2TTi 



-l)*'-^e-^ 



27r 



(5.110) 
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The second integral representation of Hermite polynomials is more familar: 



5.3.1 Asymptotics of i/j{p + q^) and i/jr'{p + q^) 

We only consider ipr' with a^' = a. ip^i with a^' < a and ip can be solved similarly, 
and we only give the results. 
We have 



where 



^ - — F^ = e "^^'^''VV (5.112) 



g{z) = -(a+ 1) ( 1 + ^ ) z + log2-7-2log(z- 1). (5.113) 



Now we can write (2.77) as 



2ot 



s 



z rfz. (5.114) 



For g{^z\ we have 

• (yf (2) = —(a + 1) 1 H — ^ H , with zero points z = ^ and 

\ 7"'a J z z — 1 1 + 7^a 



1 

z 



1 + a 

1 \ , / 1 
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Hence locally around z = 



1 + a 



w 



(1 — 7 ) log(l + a) — 7 loga + 7 ni 



^{l + af (^l-^]w^ + R2iw), (5.115) 



where 



R2{w) = as w ^ CO. 



(5.116) 



After the substitution z = w + we get by (5.114) 



1 f M((l+;^)+(l-7-2)log(l+a)+7-2loga-7-27rj+|(l+a)2(l~;^)t«2-/?2{«')) 



27ri 



(» - if.)' 



aj — a 



(l + a)(l + a,) 



2m 



W'{l + ay'-''eTf-a^ 



{^'-M (11 r + (1 + a)(l + a,) 



— a 



(5.117) 



where is a contour around w = —j^, composed of , S^, and Sf'^, which 



are defined as (g = (1 + a)^/l 



1 1 



Sf ={it - q-^/M I -2 < t < 2}, 
=|2^ _ t I g-i/M < t < 4}, 
={_4_it I -2 < t < 2}, 

Sf ={t - 2i I -4 < t < -q'^/M} 



(5.118) 
(5.119) 
(5.120) 
(5.121) 



and for the asymptotic analysis, we define 
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^2 




Figure 5.9: S 



M 



Aw e t'' I \w\ < M-^/^, 

"^remote \ local • 



(5.122) 

(5.123) 



Now if we denote 



FMaf (W) 



(1 + a) 



t' 

Ml e 



(l+a)y(l--5L^^»+l(l+a)2(l-;^)A^«;2_Mi?2(^) 



— a 



(1 + a)(l + aj) 



w 



(5.124) 



We have 



(l + a)W 1 



M 



(5.125) 



and results similar to lemmas l5.lH5.3i 
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Lemma 5.9. 



for any ^ > T. 



Lemma 5.10. 



for any ^ > T. 



27ri 



< 



1 e-€ 



3 Ml/10 ^ 



1 

27n 



yM 
remote 



FMat'{^,w)dw 



< 



1 e- 



3 Ml/10' 



Lemma 5.11. IfT is fixed and M is large enough, 



„ ., FMat'ii, W)dw - ^\ ^ "j, Cay-t'i-l) 

ZTTI ly-M — O 



for any ^ > T. 



(5.126) 



(5.127) 



e 2 



< 



1 e-? 



3 Ml/10' 
(5.128) 



Since their proofs are similar to those of lemmas 5J^-5^, we only give the proof 
of lemma 15.111 



Proof of lemma \5Jl\ On Sj^/^^j, \w\ < M'^/^, and R2{w) = 0{M"^/^), so that 



-a 



:i + a)W 1 



\ \{l + a){l + ai) 
I 'Vv « 2i u w*-^ (1 + 0(M-^/^)) 



M 



(5.129) 



and the 0(M i/^) term is independent of ^. After the substitution m = (1 
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a) J(l - Mw, we get 



27rz 



local 



FMat'{i,w)dw 



{—ay Im jy, 



S^^\u''-^du (1 + 0{M-^l^)) . (5.130) 



<(i+ 



V 7 1 



Afl/10 



On if ^ > T, |e(«-^)"| < e^e'^ and we have 



Im Jf^M {—CI') v27r 



-a 



27rz 



Til—— f' — A 

e 2u 



>(l + a),/ir^=:A/l/10 



'^''■"*'2^ 



Tv—— t' — l 



duO{M-^'^) 



(5.131) 



and we can get the result by direct calculation. 



□ 



By lemmas 5.9-5.11 and (5.125), we have the convergence resuh for ip^' with 



a,/ = a 



{l + a). 



M 



e 1+" 



{-a)^{l + aY^-^ 



-a 



271 



-e 2 



< 



Ml/10 



. (5.132) 



By the same method, we can get the convergence resuh for ipr'iO with r' = 
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1, . . . , r — t and ^ >T 



1 - 



M- 



e 1+" 



-a)^(l + a)^^-^ 

(1 + aY 



-ay 



Car' -I 



2e_il 

6 3 2 



27r 



< 



(5.133) 



and for with ^ > T 



:i + a) 



1 - 



M 



72aV i-a)^{l + a)^-^ 



,6 3 2 



-a 



27r 



-€/3 



< 



MVio- 
(5.134) 



5.3.2 Asymptotics of + qj]) and 'i/'r'lj^ + QV) 

We only consider -i/'^' with Og/ = a, and only state the results for ip,f.i with a^i < a and 
ip, since they are simpler. 



ipr'ip + Q''?) is defined in (2.78) by a contour integral with poles z = 1 and ti~, 
j = 1, . . . ,s inside the contour. By the residue theorem, the value of ipr'{p + Qv) is 
the sum of residues at these poles. We will see that the contribution of poles other 
than are negligible, and we are going to calculate the residue at To consider 
these two kinds of poles separately, we deform F into the sum of two disconnected 
contours and Frisht, where F^ includes rr- and excludes other poles, and vice 
versa for Flight- 
We have 

(5.135) 



-Myz 



{Z - 1)^ _ -Mg(z) + il+a)Ml~^)Mr,z 
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where g{z) is defined in (5.113). Then we can write (2.78) as 



2TTi 



^Mg(z)+{l+a)^ {i^:^)M7jz 



r 1 UFright 
1 + a 



+ an 



'3 \ / I 

z 



1 + a 



dz, (5.136) 



and after the substitution z = w + we get 



Mg(z)-il+a)J(l--^)Mvz {Z - 1)' 



r 1 

TFa 



yr—r'+l 



n— r 



+ a, 



1 + a 



dz 



^Af(-(l-;^) -(1-7-2) log{l+a)-7-2 log a+-y~'^7Ti-^(l+af(l-^'^w^+R2{w)^ 



i-ir 



1 ^r-r'+T I ^ 



aj — a 



(1 + a)(l + aj) 



dw 



-g 1+a' 



a^ll + a)"-" 



]^ \r— r'+l 



n 



aj — a 



(l + a)(l + a,) 



(5.137) 



where is a contour around w = 0, defined as 



r, 



M 



e*V3 



< t < 27r 



(5.138) 
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If we denote 



GMat'iV,^) = 



:i + a)W 1- 



^ 6 



pM 
^ 4 



pM 
^ 2 



pM 
^ 3 



pM 
5 ♦ 



pM 
^ 7 



Figure 5.10: and T^^t 



i-t' 



+ a)(l + Oj) 



-, (5.139) 



We have 



i-t' 



(l + a)W 1 



M 



-a)^(l + a)^-^eiT^V,,(p + g?7) = 
„— / GMat'{v,'w)dw + (p GMat'{v,'w)dw, (5.140) 



with the definition of f introduced later, and 
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Lemma 5.12. IfT is fixed and M is large enough, 



\t'-i 



1 e''/^ 



for any rj >T. 

Proof. On f |w| = M-i/^ and R2{w) = 0{M-^/^), so that 

GMat'{r],w) = 

(1 + ar-i Ull^(l + a)(l + a,) 



i-t' 



:i + a)W 1- 



M 



e 'V ^) h +C)(M"^/^)) , (5.142) 

and the 0(M~^/^) is independent oirj. After the substitution u = (l+a)^ ^1 — Mw, 



we get 



^ |„ GM.,.(,.™)rf™ = ^ £ e— (1 + O(M-V^)) . (5.143) 



\t'-i 



X /" f (Xj"^ f X 



< 



el2^/3|e'?/3 



'1 + a)'''-iCr/_t' 27ri 



duO{M-^/^) 



, (5.144) 



and we can get the result by direct calculation. 



□ 



Now we estimate the integral of Guat' over T^^^^, where T^^^^ is defined as the 
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union of Tf U ■ ■ ■ UTf , as 



M 



^■^M 



f A/ 



5 



f A/ 
6 



^Af 



^A/ 



(5 -it 



6 6 
— < t < - 
3 - - 3 



6 

-t + -t 
3 



T^a^ - 1 



5 



(l + 72a)(l + a) 
(5 < t < 



< t < -(5 



7'a2 - 1 



(l+72a)(l + a) 



1 



(l + 72a)(l + a) 
72a2 - 1 



(l + 72a)(l + a) 



it 



-t + Ai 
t-Ai 

{Cright " 



-a 



right 



< t < 



-4 < t < 



3 - - " 
-f^a^ - 1 



72a2 - 1 



(l+72a)(l + a) 
I -4 < t < 4} , 



(l + 7'a)(l 



5.145) 

5.146) 

5.147) 

5.148) 

5.149) 

5.150) 

5.151) 
5.152) 



where 5 is a very small positive number and Cright is the same large positive number 
as that in the definition of F*^. Then we can prove 



Lemma 5.13. IfT is fixed and M is large enough, 



27ri jpM 

right 



GMat'{'n^w)dw 



< 



1 

2 Ml/3 ^ 



(5.153) 



for any rj >T. 



The proof is similar to those of lemmas 5.1 and 5.2 
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Hence by lemmas 5.12 and 5.13, and (5.140), we have 



i-t' 



:i + a)W 1 



M (-a)^(l + a)*^-^eiT^^e-2''/3v;,,(p + qr]) 



-a] 



\t'-i 



{i + ay'~^Ca,r'-t' (t'-iy. 



g-'?/3 



For ipr' with ag' < a or ip, we can choose Tf^-^^ so that all poles are included in it 



and we can get the estimation by results similar to lemma 5.13 The result is 



;i + a) 



M(-a)^(l + a)^-^eifs?'e-2''/3^,(p + gr/) 



< 



e-'?/3 



(5.155) 



where ^Z'^, stands for ipr' with a^' < a, or simply ip. 



5.4 Asymptotics of Laguerre polynomials and re- 
lated functions 

In chapter 3, we need asymptotic results of L^2N^2 ^m^-i ^''^ '^2N-i, 'ipm-ii ^^c. 
They can be expressed by linear combination of Laguerre polynomials and has the 



similar integral representation. Like (2.41), we have 



45"r"»(M-) = -^1 j^—^dz, (5.156) 
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and by (3.54), we get 



2711 /r l + 

^2{M-N) 



z-l 



dz 



(1 + a)2(*^-^)+i 



z — 



l+a 



(\ + „\2{M-N)+1 2N-1 . 
\^ ^ " g2Mx 1 ^-2Mxz ^ 



2T\i 



2M 



Z - 1 



-dz. 



(2- 1)2^ ((l + a)^- 1)^ 

(5.157) 



If the pole z = is inside of F, then 



(1 + a)2(^'^-^)+i 



27ri 



e^^^ (P e 



^2{M-N) 
-2M.z^ . 



-^2Mx 



z — 



(5.158) 



l+a 



and 



(P2N-1[X) 



n _L „A2(M-Af)+1 2Ar-l . 
l-"- + "J " g2Mx i g-2Afa;z ^ 



27li 



2M 



Z-l 



-dz. 



z-1)^^ {{l + a)z-l)z 

(5.159) 



All other relevant functions can also be expressed by integrals by (2.41). We analyze 



three typical examples, and all other results can be derived similarly. 



5.4.1 L 



{2{M-N)) 
2N-1 



{2Mx) around (1 + -f-'^f 



We assume in p = (1 + 7 ) and q 



(7+1) 



4/3 



7(2M)2/3 ' ^^^^ X = p + q^. By methods in 
the analysis of ip^p + q^) and ipr'ip + qO subsection 



5.1.2 



if we denote 



^(0 



1 

2TTi 



-2Mxz 



,2M-1 



iz-1 



\2N 



dz. 



(5.160) 
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we have for ^ >T 



7 



7 



-l)Ai(0 



-5/2 



< 



jVf 1/40 • 



(5.161) 



The relation between and L^2N^i ^^\'^Mx) is 



^M-N-l/2 

-(2{M-N))i^^^^^-^^M-N-l/2-Mx_^ ^^2Mx_ 



g7 + l 



(5.162) 



For X = p + g^, ^ > T, we have the point-wise with respect to ^ 



^M-N ^M-N-1/2 



/ 7+1 \ fiT+l 



(5.163) 



and for ^ > 0, 



< 



gM-Af ^M-Af-1/2 

1 \ P 7 + 1 



< 1. 



Therefore we know that for M large enough and ^ > T 



(5.164) 



^-2^-i(^ + l)^/3(2M)i/3e^-^-^4'i'^V'^^^(2Mx)a;^^-^-i/2g-Afx _ ^^^^^ 

e-f/2 



5.4.2 '?/^2A^-i(a^) around (1 +7 when a = ^ ^ 



We still take p = {1 + 7"^)^ g 



7(2M)2/3: 



and z = p + We use the integral 



representation (5.159) of iIj2N~i{x), and we will make sure that the pole z 



l+a 



IS 



inside F when we deform it. Since = 7 ^, we can write (5.159) as 



.2M 



Z - 1 



(7 + l)2(M-iV)+l e2M. J. ^^^^^ 



dz. 

(5.166) 
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If we denote 



,2M 



-2Mxz 



z-l 



- 1)2^ ((1+ 7-1)2 -1)Z 



dz, (5.167) 



by methods in the analysis of ipr'ip + iCi in subsection 5.2.2, we have for ^ > T, 



(7 + I) 



2(M-Af)+1 



7 



2M 



< 



jVfl/40- 



(5.168) 



We can express iIj2n-i{,x) as 



X 



M-N+l/2 



g7 + l 



(^ + l)2{M-iV)+l ^ 



e^'^^^2^_i(0. (5.169) 



7 



2M 



Similar to (5.163) and (5.164), we have point wise convergence with respect to ^ 



^M-N ^M-N+l/2 



(5.170) 



and if ^ > T, then 



gM-7V ^M-N+l/2 / ^ 

< 2{M-N)+1 ^2=1Mx ^^^^ [m 



7+1 
7 



g7+l 



(5.171) 



Therefore, we have 



M-N ( 7 



7 + 1 



2(J\/-iV)+l 



^2iV-l(x)-S«(0 



< 



(5.172) 
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5.4.3 if^^^ ^^\2Mx) around (1 + a) (l + when a > 7"^ 

We take p = {1 + a) (l + q = (1 + a)^l - and x = p + qC Like 



(5.160), we define 



,2M-1 



~2Mxz 



\2N 



dz. 



With g{z) defined in (5.113), we have 



(5.173) 



~ 1 / 2Mg{z)-{l+a)Ml--^)2Mizdz 



(5.174) 



After the substitution z = w + i1^2a, i we get similar to (5.117) 



2Mg{z)-(l+a)^ [l- ^^'^2M£,z dz 

Z 

-2A/((l+a) + (l-7-2)log(l+72a)--log(762a-7-27rii{7-i+7a)2(l-^)«)2-iJ3(«,)j 



{l+a)J{l~^)2Md^+^\ dw 



l+7^a 



2Mx 



(72a)2*V^T^ 
(1 +72a)2(A^-JV) 



pA/ 



1+7^0 



(5.175) 



where 



Rsiw) = 0{w^), as w ^ 0, 



(5.176) 



and is a contour around w = piji^, composed of F^, , T^^ and r4^, which are 



CHAPTER 5. ASYMPTOTIC ANALYSIS 



133 



defined as 



f f ={-it + g"V(2M) I -2 < t < 2}, 



rf ={4 - t + 2i I < t < 4 - q-'/M} 



rf ={A + it \ -2<t< 2}, 

ff ={t - 2i I g"V(2M) < t < 4}. 



(5.177) 
(5.178) 

(5.179) 
(5.180) 



By methods in the analysis of ipr'{p + qO subsection 5.3.1, we have for ^ > T 




Figure 5.11: F 



M 



(1 + 72a)2(^-^) 



(^a) 



2M 



v/(72a2 - l)2Me^^^ V^„(0 



-1 _ 1 / 7+7" 



7+70 A ' 



< 



Ml/10- 



(5.181) 



We have the result similar to (5.162) 



■Mx X 



M-N-1/2 ^ 



(l + a) 1+^ 

e V 7 0/ 
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For X — p + q^, ^ >T, we have the pointwise convergence with respect to ^ 



-,M-N 



M^oo 



X 



M-N-l/2 



((1 + a) (l + ^)) 



M-N-l/2 



— e 



1 (7 

'4 (7^a+l)^ 



1)(7''-I) f2 



(5.183) 



(l+a) 1+4- 
e V 7 a 



and for ^ > 0, 



< 



a; 



M-Af-l/2 



((1 + a) (l+^)) 



M- AT- 1/2 



< 1. 



(5.184) 



(l + o)( 1+4- 

e v 7 



Therefore if M is large enough, 



(7^0 + l)M-iV+l/2gM-iV^(^2a2 _ 1)2M 



7^a+a+2 



-V2(a + l)A^-JV-i/2 



-1 



Mx 



1 7^a^+7-^a^+47^a+7^ + l c2 



(7^a+l)^ 



< 



MVio- 



(5.185) 
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